[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:46:01 dist-test-slave-lkn4 krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:46:01 dist-test-slave-lkn4 krb5kdc[399](info): set up 2 sockets
Sep 27 12:46:01 dist-test-slave-lkn4 krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260927 12:46:02.155763 392 mini_kdc.cc:121] Time spent starting KDC: real 0.957s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/krb5kdc/test-user.keytab.
Sep 27 12:46:02 dist-test-slave-lkn4 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:46:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:46:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:46:02.274158 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40365
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40365
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:46:02.381811 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:46:02.382108 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:46:02.382164 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:46:02.386081 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:46:02.386137 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:46:02.386163 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:46:02.386184 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:46:02.386205 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:46:02.391325 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40365
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:46:02.392447 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:46:02.393445 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 12:46:02.403401 412 server_base.cc:1034] running on GCE node
W20260927 12:46:02.403899 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:02.403991 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:02.403898 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:02.404693 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:46:02.405789 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:46:02.406936 412 hybrid_clock.cc:648] HybridClock initialized: now 1790513162406884 us; error 64 us; skew 500 ppm
Sep 27 12:46:02 dist-test-slave-lkn4 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513162, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:46:02.409574 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:46:02.410586 412 webserver.cc:460] Webserver started at http://127.0.98.62:43365/ using document root <none> and password file <none>
I20260927 12:46:02.411123 412 fs_manager.cc:359] Metadata directory not provided
I20260927 12:46:02.411185 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:46:02.411386 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:46:02.413467 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/minicluster-data/master-0/data/instance:
uuid: "e063fdc12fc74e5e8367ca4c9e4f2c0b"
format_stamp: "Formatted at 2026-09-27 12:46:02 on dist-test-slave-lkn4"
server_key: "fefb49960e2b74f3d1c30d36c44b4882"
server_key_iv: "69db45fcee62f708d5e535296091fb85"
server_key_version: "encryptionkey@0"
I20260927 12:46:02.414018 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/minicluster-data/master-0/wal/instance:
uuid: "e063fdc12fc74e5e8367ca4c9e4f2c0b"
format_stamp: "Formatted at 2026-09-27 12:46:02 on dist-test-slave-lkn4"
server_key: "fefb49960e2b74f3d1c30d36c44b4882"
server_key_iv: "69db45fcee62f708d5e535296091fb85"
server_key_version: "encryptionkey@0"
I20260927 12:46:02.417862 412 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 12:46:02.420401 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:02.421742 412 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:46:02.421865 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/minicluster-data/master-0/wal
uuid: "e063fdc12fc74e5e8367ca4c9e4f2c0b"
format_stamp: "Formatted at 2026-09-27 12:46:02 on dist-test-slave-lkn4"
server_key: "fefb49960e2b74f3d1c30d36c44b4882"
server_key_iv: "69db45fcee62f708d5e535296091fb85"
server_key_version: "encryptionkey@0"
I20260927 12:46:02.421979 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:46:02.435866 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:46:02.439059 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:46:02.439224 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:46:02.445771 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40365
I20260927 12:46:02.445823 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40365 every 8 connection(s)
I20260927 12:46:02.446745 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:46:02.449532 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:46:02.450526 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 412
I20260927 12:46:02.450699 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/minicluster-data/master-0/wal/instance
I20260927 12:46:02.452297 392 external_mini_cluster.cc:1428] Setting key d4d163bc24015ed9fbe9271cee6162a8
I20260927 12:46:02.456458 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e063fdc12fc74e5e8367ca4c9e4f2c0b: Bootstrap starting.
I20260927 12:46:02.459214 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e063fdc12fc74e5e8367ca4c9e4f2c0b: Neither blocks nor log segments found. Creating new log.
I20260927 12:46:02.460096 480 log.cc:824] T 00000000000000000000000000000000 P e063fdc12fc74e5e8367ca4c9e4f2c0b: Log is configured to *not* fsync() on all Append() calls
I20260927 12:46:02.462069 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e063fdc12fc74e5e8367ca4c9e4f2c0b: No bootstrap required, opened a new log
I20260927 12:46:02.465745 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P e063fdc12fc74e5e8367ca4c9e4f2c0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e063fdc12fc74e5e8367ca4c9e4f2c0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40365 } }
I20260927 12:46:02.465955 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P e063fdc12fc74e5e8367ca4c9e4f2c0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:46:02.465998 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P e063fdc12fc74e5e8367ca4c9e4f2c0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e063fdc12fc74e5e8367ca4c9e4f2c0b, State: Initialized, Role: FOLLOWER
I20260927 12:46:02.466462 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P e063fdc12fc74e5e8367ca4c9e4f2c0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e063fdc12fc74e5e8367ca4c9e4f2c0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40365 } }
I20260927 12:46:02.466622 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P e063fdc12fc74e5e8367ca4c9e4f2c0b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:46:02.466697 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P e063fdc12fc74e5e8367ca4c9e4f2c0b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:46:02.466795 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e063fdc12fc74e5e8367ca4c9e4f2c0b [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:46:02.468457 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P e063fdc12fc74e5e8367ca4c9e4f2c0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e063fdc12fc74e5e8367ca4c9e4f2c0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40365 } }
I20260927 12:46:02.468796 480 leader_election.cc:302] T 00000000000000000000000000000000 P e063fdc12fc74e5e8367ca4c9e4f2c0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e063fdc12fc74e5e8367ca4c9e4f2c0b; no voters:
I20260927 12:46:02.469146 480 leader_election.cc:288] T 00000000000000000000000000000000 P e063fdc12fc74e5e8367ca4c9e4f2c0b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:46:02.469211 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e063fdc12fc74e5e8367ca4c9e4f2c0b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:46:02.469435 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P e063fdc12fc74e5e8367ca4c9e4f2c0b [term 1 LEADER]: Becoming Leader. State: Replica: e063fdc12fc74e5e8367ca4c9e4f2c0b, State: Running, Role: LEADER
I20260927 12:46:02.469754 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P e063fdc12fc74e5e8367ca4c9e4f2c0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e063fdc12fc74e5e8367ca4c9e4f2c0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40365 } }
I20260927 12:46:02.470510 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P e063fdc12fc74e5e8367ca4c9e4f2c0b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:46:02.471454 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P e063fdc12fc74e5e8367ca4c9e4f2c0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e063fdc12fc74e5e8367ca4c9e4f2c0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e063fdc12fc74e5e8367ca4c9e4f2c0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40365 } } }
I20260927 12:46:02.471586 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P e063fdc12fc74e5e8367ca4c9e4f2c0b [sys.catalog]: This master's current role is: LEADER
I20260927 12:46:02.471897 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P e063fdc12fc74e5e8367ca4c9e4f2c0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e063fdc12fc74e5e8367ca4c9e4f2c0b. Latest consensus state: current_term: 1 leader_uuid: "e063fdc12fc74e5e8367ca4c9e4f2c0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e063fdc12fc74e5e8367ca4c9e4f2c0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40365 } } }
I20260927 12:46:02.471984 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P e063fdc12fc74e5e8367ca4c9e4f2c0b [sys.catalog]: This master's current role is: LEADER
I20260927 12:46:02.472651 494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 27 12:46:02 dist-test-slave-lkn4 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:46:02.475883 494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:46:02.479558 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:02.456124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57178 (local address 127.0.98.62:40365)
0927 12:46:02.456634 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:02.456654 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:02.457596 (+ 942us) server_negotiation.cc:407] Connection header received
0927 12:46:02.461495 (+ 3899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:02.461510 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:02.461857 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:02.462178 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:02.465008 (+ 2830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:02.466420 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:02.468236 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:02.468739 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:02.468865 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:02.474100 (+ 5235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:02.474108 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:02.474127 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:02.474158 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:02.475846 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:02.476432 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:02.476438 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:02.476442 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:02.476552 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:02.476898 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:02.476902 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:02.476904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:02.477298 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:02.477482 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:02.478245 (+ 763us) server_negotiation.cc:299] Negotiation successful
0927 12:46:02.478562 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":123,"threads_started":1}
I20260927 12:46:02.481873 494 catalog_manager.cc:1347] Generated new cluster ID: 9b7f8b5b54cd4b428f5cbc57e0fbe360
I20260927 12:46:02.481936 494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:46:02.493412 494 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:46:02.494400 494 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:46:02.499758 494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e063fdc12fc74e5e8367ca4c9e4f2c0b: Generated new TSK 0
I20260927 12:46:02.500438 494 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 12:46:02.532966 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40365
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:46:02.662571 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:46:02.662837 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:46:02.662887 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:46:02.666749 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:46:02.666805 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:46:02.666863 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:46:02.671978 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40365
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:46:02.672972 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:46:02.673902 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:46:02.682996 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:02.683035 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:02.683022 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:02.683126 506 server_base.cc:1034] running on GCE node
I20260927 12:46:02.683626 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:46:02.684258 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:46:02.685434 506 hybrid_clock.cc:648] HybridClock initialized: now 1790513162685399 us; error 49 us; skew 500 ppm
Sep 27 12:46:02 dist-test-slave-lkn4 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513162, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:46:02.688529 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 12:46:02.689705 506 webserver.cc:460] Webserver started at http://127.0.98.1:42417/ using document root <none> and password file <none>
I20260927 12:46:02.690353 506 fs_manager.cc:359] Metadata directory not provided
I20260927 12:46:02.690472 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:46:02.690748 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:46:02.692839 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/minicluster-data/ts-0/data/instance:
uuid: "40f04de5c63f484baa090008386bc61d"
format_stamp: "Formatted at 2026-09-27 12:46:02 on dist-test-slave-lkn4"
server_key: "f7b13c1f8c40476ceb1f7365f6a64702"
server_key_iv: "2f9b9e3be518c4533eaa4a551f879fb8"
server_key_version: "encryptionkey@0"
I20260927 12:46:02.693387 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/minicluster-data/ts-0/wal/instance:
uuid: "40f04de5c63f484baa090008386bc61d"
format_stamp: "Formatted at 2026-09-27 12:46:02 on dist-test-slave-lkn4"
server_key: "f7b13c1f8c40476ceb1f7365f6a64702"
server_key_iv: "2f9b9e3be518c4533eaa4a551f879fb8"
server_key_version: "encryptionkey@0"
I20260927 12:46:02.697091 506 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 12:46:02.699546 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:02.700709 506 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 12:46:02.700846 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/minicluster-data/ts-0/wal
uuid: "40f04de5c63f484baa090008386bc61d"
format_stamp: "Formatted at 2026-09-27 12:46:02 on dist-test-slave-lkn4"
server_key: "f7b13c1f8c40476ceb1f7365f6a64702"
server_key_iv: "2f9b9e3be518c4533eaa4a551f879fb8"
server_key_version: "encryptionkey@0"
I20260927 12:46:02.700941 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:46:02.720500 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:46:02.723547 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:46:02.723727 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:46:02.724303 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:46:02.725124 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:46:02.725174 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:02.725229 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:46:02.725256 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:02.734212 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42499
I20260927 12:46:02.734251 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42499 every 8 connection(s)
I20260927 12:46:02.735370 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:46:02.740131 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 506
I20260927 12:46:02.740222 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:46:02.740460 392 external_mini_cluster.cc:1428] Setting key dd9b1635a66a6d46c135594fdc8c6d28
Sep 27 12:46:02 dist-test-slave-lkn4 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513162, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:46:02.749518 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:02.737111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58635 (local address 127.0.98.62:40365)
0927 12:46:02.737243 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:02.737248 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:02.738097 (+ 849us) server_negotiation.cc:407] Connection header received
0927 12:46:02.739385 (+ 1288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:02.739387 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:02.739430 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:02.739498 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:02.740962 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:02.741544 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:02.742814 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:02.743140 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:02.743245 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:02.745950 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:02.745953 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:02.745956 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:02.745978 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:02.747410 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:02.748000 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:02.748003 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:02.748005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:02.748080 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:02.748437 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:02.748439 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:02.748440 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:02.748608 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:02.748677 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:02.749295 (+ 618us) server_negotiation.cc:299] Negotiation successful
0927 12:46:02.749398 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 12:46:02.750537 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:02.737524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40365 (local address 127.0.98.1:58635)
0927 12:46:02.737938 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:02.737978 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:02.739201 (+ 1223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:02.739636 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:02.739642 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:02.740135 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:02.740769 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:02.740787 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:02.741676 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:02.741680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:02.742688 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:02.742698 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:02.743306 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:02.743311 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:02.743506 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:02.744122 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:02.744143 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:02.745788 (+ 1645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:02.747527 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:02.747533 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:02.747551 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:02.747893 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:02.748172 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:02.748175 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:02.748177 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:02.748356 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:02.748701 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:02.748709 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:02.749038 (+ 329us) client_negotiation.cc:769] Sending connection context
0927 12:46:02.749278 (+ 240us) client_negotiation.cc:240] Negotiation successful
0927 12:46:02.749629 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":91,"threads_started":1}
I20260927 12:46:02.751775 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40365
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 12:46:02.752053 635 heartbeater.cc:464] Registering TS with master...
I20260927 12:46:02.752557 635 heartbeater.cc:511] Master 127.0.98.62:40365 requested a full tablet report, sending...
I20260927 12:46:02.754086 444 ts_manager.cc:194] Registered new tserver with Master: 40f04de5c63f484baa090008386bc61d (127.0.98.1:42499)
I20260927 12:46:02.755178 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58635
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 12:46:02.766564 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40365
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:46:02.878252 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:46:02.878523 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:46:02.878588 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:46:02.882700 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:46:02.882767 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:46:02.882846 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:46:02.888144 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40365
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:46:02.889197 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:46:02.890300 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 12:46:02.900195 640 server_base.cc:1034] running on GCE node
W20260927 12:46:02.900142 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:02.900179 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:02.900147 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:02.900810 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:46:02.901443 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:46:02.902593 640 hybrid_clock.cc:648] HybridClock initialized: now 1790513162902579 us; error 33 us; skew 500 ppm
Sep 27 12:46:02 dist-test-slave-lkn4 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513162, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:46:02.905216 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 12:46:02.906157 640 webserver.cc:460] Webserver started at http://127.0.98.2:33491/ using document root <none> and password file <none>
I20260927 12:46:02.906669 640 fs_manager.cc:359] Metadata directory not provided
I20260927 12:46:02.906725 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:46:02.906898 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:46:02.908882 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/minicluster-data/ts-1/data/instance:
uuid: "7cc83336ec1c4b768474b513b13131c3"
format_stamp: "Formatted at 2026-09-27 12:46:02 on dist-test-slave-lkn4"
server_key: "9a8eceb2a87af1838facc120c56233b4"
server_key_iv: "d4aa8810a3d5de0464f832bbd43d8e98"
server_key_version: "encryptionkey@0"
I20260927 12:46:02.909462 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7cc83336ec1c4b768474b513b13131c3"
format_stamp: "Formatted at 2026-09-27 12:46:02 on dist-test-slave-lkn4"
server_key: "9a8eceb2a87af1838facc120c56233b4"
server_key_iv: "d4aa8810a3d5de0464f832bbd43d8e98"
server_key_version: "encryptionkey@0"
I20260927 12:46:02.913187 640 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260927 12:46:02.915586 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:02.916857 640 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 12:46:02.916949 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/minicluster-data/ts-1/wal
uuid: "7cc83336ec1c4b768474b513b13131c3"
format_stamp: "Formatted at 2026-09-27 12:46:02 on dist-test-slave-lkn4"
server_key: "9a8eceb2a87af1838facc120c56233b4"
server_key_iv: "d4aa8810a3d5de0464f832bbd43d8e98"
server_key_version: "encryptionkey@0"
I20260927 12:46:02.917034 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:46:02.931597 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:46:02.934648 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:46:02.934809 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:46:02.935324 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:46:02.936072 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:46:02.936116 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:02.936153 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:46:02.936172 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:02.945586 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44053
I20260927 12:46:02.945613 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44053 every 8 connection(s)
I20260927 12:46:02.946595 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:46:02.952519 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 640
I20260927 12:46:02.952631 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:46:02.952874 392 external_mini_cluster.cc:1428] Setting key b0a4e4988250dba9a586eb0aef48199e
Sep 27 12:46:02 dist-test-slave-lkn4 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513162, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:46:02.960227 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:02.948413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42455 (local address 127.0.98.62:40365)
0927 12:46:02.948556 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:02.948561 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:02.949432 (+ 871us) server_negotiation.cc:407] Connection header received
0927 12:46:02.950744 (+ 1312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:02.950747 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:02.950795 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:02.950868 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:02.952019 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:02.952540 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:02.953875 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:02.954144 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:02.954232 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:02.956963 (+ 2731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:02.956967 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:02.956969 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:02.956991 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:02.958367 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:02.958815 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:02.958819 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:02.958821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:02.958895 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:02.959193 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:02.959196 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:02.959198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:02.959388 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:02.959471 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:02.959954 (+ 483us) server_negotiation.cc:299] Negotiation successful
0927 12:46:02.960068 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 12:46:02.961078 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:02.948806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40365 (local address 127.0.98.2:42455)
0927 12:46:02.949282 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:02.949325 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:02.950544 (+ 1219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:02.951024 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:02.951031 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:02.951413 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:02.951904 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:02.951918 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:02.952662 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:02.952665 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:02.953733 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:02.953748 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:02.954267 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:02.954271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:02.954433 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:02.955019 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:02.955035 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:02.956790 (+ 1755us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:02.958463 (+ 1673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:02.958469 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:02.958484 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:02.958727 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:02.958970 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:02.958973 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:02.958975 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:02.959113 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:02.959480 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:02.959487 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:02.959727 (+ 240us) client_negotiation.cc:769] Sending connection context
0927 12:46:02.959897 (+ 170us) client_negotiation.cc:240] Negotiation successful
0927 12:46:02.960214 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":104,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 12:46:02.962224 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40365
I20260927 12:46:02.962467 769 heartbeater.cc:464] Registering TS with master...
I20260927 12:46:02.962960 769 heartbeater.cc:511] Master 127.0.98.62:40365 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 12:46:02.963991 444 ts_manager.cc:194] Registered new tserver with Master: 7cc83336ec1c4b768474b513b13131c3 (127.0.98.2:44053)
I20260927 12:46:02.964524 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42455
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 12:46:02.979013 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40365
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:46:03.088963 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:46:03.089248 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:46:03.089308 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:46:03.093470 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:46:03.093535 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:46:03.093617 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:46:03.099004 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40365
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:46:03.100065 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:46:03.101115 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:46:03.110316 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:03.110342 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:03.110538 774 server_base.cc:1034] running on GCE node
W20260927 12:46:03.110316 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:03.111076 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:46:03.111665 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:46:03.112833 774 hybrid_clock.cc:648] HybridClock initialized: now 1790513163112809 us; error 38 us; skew 500 ppm
Sep 27 12:46:03 dist-test-slave-lkn4 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513163, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:46:03.115518 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 12:46:03.116402 774 webserver.cc:460] Webserver started at http://127.0.98.3:39147/ using document root <none> and password file <none>
I20260927 12:46:03.116899 774 fs_manager.cc:359] Metadata directory not provided
I20260927 12:46:03.116953 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:46:03.117107 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:46:03.119125 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/minicluster-data/ts-2/data/instance:
uuid: "51cce9f1b4c3452c9916d0e46b51aae2"
format_stamp: "Formatted at 2026-09-27 12:46:03 on dist-test-slave-lkn4"
server_key: "8dfc30201744ad5764487dd0e63f6279"
server_key_iv: "d2b24f22ceb4fe8aa4d165ab260fab51"
server_key_version: "encryptionkey@0"
I20260927 12:46:03.119638 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/minicluster-data/ts-2/wal/instance:
uuid: "51cce9f1b4c3452c9916d0e46b51aae2"
format_stamp: "Formatted at 2026-09-27 12:46:03 on dist-test-slave-lkn4"
server_key: "8dfc30201744ad5764487dd0e63f6279"
server_key_iv: "d2b24f22ceb4fe8aa4d165ab260fab51"
server_key_version: "encryptionkey@0"
I20260927 12:46:03.123428 774 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 12:46:03.125898 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:03.127182 774 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:46:03.127274 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/minicluster-data/ts-2/wal
uuid: "51cce9f1b4c3452c9916d0e46b51aae2"
format_stamp: "Formatted at 2026-09-27 12:46:03 on dist-test-slave-lkn4"
server_key: "8dfc30201744ad5764487dd0e63f6279"
server_key_iv: "d2b24f22ceb4fe8aa4d165ab260fab51"
server_key_version: "encryptionkey@0"
I20260927 12:46:03.127394 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:46:03.145169 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:46:03.148061 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:46:03.148218 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:46:03.148780 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:46:03.149549 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:46:03.149591 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:03.149628 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:46:03.149644 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:03.159241 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35751
I20260927 12:46:03.159292 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35751 every 8 connection(s)
I20260927 12:46:03.160121 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:46:03.164892 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 774
I20260927 12:46:03.165007 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790513161160093-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:46:03.165256 392 external_mini_cluster.cc:1428] Setting key a7d61a0a3d6e877d4e6257facc154853
Sep 27 12:46:03 dist-test-slave-lkn4 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513163, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:46:03.173637 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:03.161990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56907 (local address 127.0.98.62:40365)
0927 12:46:03.162141 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:03.162146 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:03.163015 (+ 869us) server_negotiation.cc:407] Connection header received
0927 12:46:03.164334 (+ 1319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:03.164336 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:03.164383 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:03.164452 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:03.165961 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:03.166546 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:03.167866 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:03.168074 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:03.168157 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:03.170424 (+ 2267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:03.170427 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:03.170429 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:03.170453 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:03.171767 (+ 1314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:03.172256 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:03.172260 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:03.172262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:03.172335 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:03.172636 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:03.172639 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:03.172641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:03.172814 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:03.172893 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:03.173356 (+ 463us) server_negotiation.cc:299] Negotiation successful
0927 12:46:03.173480 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 12:46:03.174425 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:03.162401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40365 (local address 127.0.98.3:56907)
0927 12:46:03.162851 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:03.162905 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:03.164132 (+ 1227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:03.164614 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:03.164624 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:03.165151 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:03.165823 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:03.165840 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:03.166660 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:03.166663 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:03.167758 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:03.167767 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:03.168199 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:03.168203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:03.168364 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:03.168861 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:03.168876 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:03.170274 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:03.171899 (+ 1625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:03.171905 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:03.171922 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:03.172171 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:03.172424 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:03.172427 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:03.172429 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:03.172555 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:03.172903 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:03.172909 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:03.173143 (+ 234us) client_negotiation.cc:769] Sending connection context
0927 12:46:03.173310 (+ 167us) client_negotiation.cc:240] Negotiation successful
0927 12:46:03.173610 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":100,"threads_started":1}
I20260927 12:46:03.175415 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40365
I20260927 12:46:03.175704 903 heartbeater.cc:464] Registering TS with master...
I20260927 12:46:03.176175 903 heartbeater.cc:511] Master 127.0.98.62:40365 requested a full tablet report, sending...
I20260927 12:46:03.177095 444 ts_manager.cc:194] Registered new tserver with Master: 51cce9f1b4c3452c9916d0e46b51aae2 (127.0.98.3:35751)
I20260927 12:46:03.177608 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56907
I20260927 12:46:03.179278 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 12:46:03 dist-test-slave-lkn4 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513163, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 12:46:03 dist-test-slave-lkn4 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513163, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 12:46:03.204519 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:03.195157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45740 (local address 127.0.98.1:42499)
0927 12:46:03.195493 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:03.195503 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:03.195528 (+ 25us) server_negotiation.cc:407] Connection header received
0927 12:46:03.195579 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:03.195584 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:03.195841 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:03.195963 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:03.196606 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:03.197548 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:03.198419 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:03.198811 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:03.198917 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:03.201225 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:03.201234 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:03.201247 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:03.201272 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:03.202862 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:03.203250 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:03.203256 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:03.203257 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:03.203345 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:03.203613 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:03.203616 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:03.203618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:03.203911 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:03.204100 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:03.204235 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 12:46:03.204370 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":120,"threads_started":1}
W20260927 12:46:03.205547 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45740
Sep 27 12:46:03 dist-test-slave-lkn4 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 12:46:03 dist-test-slave-lkn4 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 12:46:03.230552 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:03.222101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45748 (local address 127.0.98.1:42499)
0927 12:46:03.222279 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:03.222283 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:03.222379 (+ 96us) server_negotiation.cc:407] Connection header received
0927 12:46:03.222491 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:03.222494 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:03.222537 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:03.222596 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:03.223309 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:03.223832 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:03.224728 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:03.224946 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:03.225033 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:03.227520 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:03.227523 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:03.227525 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:03.227550 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:03.228941 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:03.229385 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:03.229388 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:03.229390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:03.229474 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:03.229754 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:03.229758 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:03.229759 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:03.229929 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:03.230031 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:03.230328 (+ 297us) server_negotiation.cc:299] Negotiation successful
0927 12:46:03.230430 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 12:46:03.231684 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 506
I20260927 12:46:03.237084 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 640
I20260927 12:46:03.243391 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 774
I20260927 12:46:03.247865 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 412
2026-09-27T12:46:03Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2079 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:46:03 dist-test-slave-lkn4 krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:46:03 dist-test-slave-lkn4 krb5kdc[917](info): set up 2 sockets
Sep 27 12:46:03 dist-test-slave-lkn4 krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260927 12:46:04.201550 392 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/krb5kdc/test-user.keytab.
Sep 27 12:46:04 dist-test-slave-lkn4 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:46:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:46:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:46:04.313136 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36205
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36205
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:46:04.420069 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:46:04.420358 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:46:04.420398 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:46:04.424320 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:46:04.424417 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:46:04.424440 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:46:04.424456 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:46:04.424470 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:46:04.429817 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36205
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:46:04.430919 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:46:04.431900 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:46:04.440210 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:04.440233 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:04.440299 930 server_base.cc:1034] running on GCE node
W20260927 12:46:04.440227 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:04.440953 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:46:04.442065 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:46:04.443235 930 hybrid_clock.cc:648] HybridClock initialized: now 1790513164443205 us; error 43 us; skew 500 ppm
Sep 27 12:46:04 dist-test-slave-lkn4 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513164, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:46:04.445976 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:46:04.446942 930 webserver.cc:460] Webserver started at http://127.0.98.62:46813/ using document root <none> and password file <none>
I20260927 12:46:04.447477 930 fs_manager.cc:359] Metadata directory not provided
I20260927 12:46:04.447536 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:46:04.447721 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:46:04.449750 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/minicluster-data/master-0/data/instance:
uuid: "9754decfc2384642b40167a7e274844c"
format_stamp: "Formatted at 2026-09-27 12:46:04 on dist-test-slave-lkn4"
server_key: "b669dd20f9d6bffa907853873ce3ef15"
server_key_iv: "ac87758b02d5e2897a0805c95e5dcd87"
server_key_version: "encryptionkey@0"
I20260927 12:46:04.450311 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/minicluster-data/master-0/wal/instance:
uuid: "9754decfc2384642b40167a7e274844c"
format_stamp: "Formatted at 2026-09-27 12:46:04 on dist-test-slave-lkn4"
server_key: "b669dd20f9d6bffa907853873ce3ef15"
server_key_iv: "ac87758b02d5e2897a0805c95e5dcd87"
server_key_version: "encryptionkey@0"
I20260927 12:46:04.453948 930 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 12:46:04.456319 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:04.457607 930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:46:04.457712 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/minicluster-data/master-0/wal
uuid: "9754decfc2384642b40167a7e274844c"
format_stamp: "Formatted at 2026-09-27 12:46:04 on dist-test-slave-lkn4"
server_key: "b669dd20f9d6bffa907853873ce3ef15"
server_key_iv: "ac87758b02d5e2897a0805c95e5dcd87"
server_key_version: "encryptionkey@0"
I20260927 12:46:04.457840 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:46:04.476310 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:46:04.479434 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:46:04.479609 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:46:04.486610 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36205
I20260927 12:46:04.486634 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36205 every 8 connection(s)
I20260927 12:46:04.487572 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:46:04.488826 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 930
I20260927 12:46:04.488916 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/minicluster-data/master-0/wal/instance
I20260927 12:46:04.489151 392 external_mini_cluster.cc:1428] Setting key 9c43f70ad3fc95d0ba5279ad16c9c53f
I20260927 12:46:04.490707 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:46:04.497323 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9754decfc2384642b40167a7e274844c: Bootstrap starting.
Sep 27 12:46:04 dist-test-slave-lkn4 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:46:04.499934 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9754decfc2384642b40167a7e274844c: Neither blocks nor log segments found. Creating new log.
I20260927 12:46:04.500844 998 log.cc:824] T 00000000000000000000000000000000 P 9754decfc2384642b40167a7e274844c: Log is configured to *not* fsync() on all Append() calls
I20260927 12:46:04.503264 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9754decfc2384642b40167a7e274844c: No bootstrap required, opened a new log
I20260927 12:46:04.504585 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:04.490733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39710 (local address 127.0.98.62:36205)
0927 12:46:04.491165 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:04.491184 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:04.491227 (+ 43us) server_negotiation.cc:407] Connection header received
0927 12:46:04.492043 (+ 816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:04.492066 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:04.492535 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:04.492899 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:04.493522 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:04.495097 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:04.496038 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:04.496751 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:04.496911 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:04.499337 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:04.499344 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:04.499360 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:04.499402 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:04.501414 (+ 2012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:04.501955 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:04.501962 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:04.501968 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:04.502136 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:04.502387 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:04.502390 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:04.502392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:04.502918 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:04.503098 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:04.503373 (+ 275us) server_negotiation.cc:299] Negotiation successful
0927 12:46:04.503667 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":135,"threads_started":1}
I20260927 12:46:04.506942 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9754decfc2384642b40167a7e274844c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9754decfc2384642b40167a7e274844c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36205 } }
I20260927 12:46:04.507226 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9754decfc2384642b40167a7e274844c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:46:04.507287 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9754decfc2384642b40167a7e274844c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9754decfc2384642b40167a7e274844c, State: Initialized, Role: FOLLOWER
I20260927 12:46:04.507689 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9754decfc2384642b40167a7e274844c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9754decfc2384642b40167a7e274844c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36205 } }
I20260927 12:46:04.507825 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9754decfc2384642b40167a7e274844c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:46:04.507884 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9754decfc2384642b40167a7e274844c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:46:04.507973 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9754decfc2384642b40167a7e274844c [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:46:04.508854 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9754decfc2384642b40167a7e274844c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9754decfc2384642b40167a7e274844c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36205 } }
I20260927 12:46:04.509173 998 leader_election.cc:302] T 00000000000000000000000000000000 P 9754decfc2384642b40167a7e274844c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9754decfc2384642b40167a7e274844c; no voters:
I20260927 12:46:04.509457 998 leader_election.cc:288] T 00000000000000000000000000000000 P 9754decfc2384642b40167a7e274844c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:46:04.509593 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9754decfc2384642b40167a7e274844c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:46:04.509784 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9754decfc2384642b40167a7e274844c [term 1 LEADER]: Becoming Leader. State: Replica: 9754decfc2384642b40167a7e274844c, State: Running, Role: LEADER
I20260927 12:46:04.510102 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9754decfc2384642b40167a7e274844c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9754decfc2384642b40167a7e274844c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36205 } }
I20260927 12:46:04.510634 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9754decfc2384642b40167a7e274844c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:46:04.511662 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9754decfc2384642b40167a7e274844c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9754decfc2384642b40167a7e274844c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9754decfc2384642b40167a7e274844c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36205 } } }
I20260927 12:46:04.511780 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9754decfc2384642b40167a7e274844c [sys.catalog]: This master's current role is: LEADER
I20260927 12:46:04.512290 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9754decfc2384642b40167a7e274844c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9754decfc2384642b40167a7e274844c. Latest consensus state: current_term: 1 leader_uuid: "9754decfc2384642b40167a7e274844c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9754decfc2384642b40167a7e274844c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36205 } } }
I20260927 12:46:04.512396 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9754decfc2384642b40167a7e274844c [sys.catalog]: This master's current role is: LEADER
W20260927 12:46:04.514943 1018 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9754decfc2384642b40167a7e274844c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 12:46:04.515058 1018 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 12:46:04.515143 1012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:46:04.515980 1012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:46:04.521811 1012 catalog_manager.cc:1347] Generated new cluster ID: e69e97253af44c6bb1e540363b59c206
I20260927 12:46:04.521868 1012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:46:04.548717 1012 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:46:04.549683 1012 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:46:04.556478 1012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9754decfc2384642b40167a7e274844c: Generated new TSK 0
I20260927 12:46:04.557299 1012 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 12:46:04.592861 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36205
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:46:04.704373 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:46:04.704658 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:46:04.704704 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:46:04.709167 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:46:04.709251 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:46:04.709318 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:46:04.715458 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36205
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:46:04.716662 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:46:04.717841 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:46:04.730571 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:04.730607 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:04.730479 1024 server_base.cc:1034] running on GCE node
W20260927 12:46:04.730871 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:04.731321 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:46:04.732031 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:46:04.733199 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790513164733156 us; error 59 us; skew 500 ppm
Sep 27 12:46:04 dist-test-slave-lkn4 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513164, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:46:04.736181 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 12:46:04.737103 1024 webserver.cc:460] Webserver started at http://127.0.98.1:42227/ using document root <none> and password file <none>
I20260927 12:46:04.737664 1024 fs_manager.cc:359] Metadata directory not provided
I20260927 12:46:04.737722 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:46:04.737892 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:46:04.740064 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/minicluster-data/ts-0/data/instance:
uuid: "1ff70c61dd8d402784b6ca7d80f45194"
format_stamp: "Formatted at 2026-09-27 12:46:04 on dist-test-slave-lkn4"
server_key: "2618d5b6ed281bd2f53b535922abc2bb"
server_key_iv: "226f8b3fba8470ac5ec93905d08bc14a"
server_key_version: "encryptionkey@0"
I20260927 12:46:04.740585 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1ff70c61dd8d402784b6ca7d80f45194"
format_stamp: "Formatted at 2026-09-27 12:46:04 on dist-test-slave-lkn4"
server_key: "2618d5b6ed281bd2f53b535922abc2bb"
server_key_iv: "226f8b3fba8470ac5ec93905d08bc14a"
server_key_version: "encryptionkey@0"
I20260927 12:46:04.745126 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 12:46:04.747807 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:04.749209 1024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 12:46:04.749326 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/minicluster-data/ts-0/wal
uuid: "1ff70c61dd8d402784b6ca7d80f45194"
format_stamp: "Formatted at 2026-09-27 12:46:04 on dist-test-slave-lkn4"
server_key: "2618d5b6ed281bd2f53b535922abc2bb"
server_key_iv: "226f8b3fba8470ac5ec93905d08bc14a"
server_key_version: "encryptionkey@0"
I20260927 12:46:04.749415 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:46:04.765148 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:46:04.768064 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:46:04.768215 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:46:04.768785 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:46:04.769591 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:46:04.769636 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:04.769676 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:46:04.769696 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:04.777988 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45295
I20260927 12:46:04.778065 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45295 every 8 connection(s)
I20260927 12:46:04.778937 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:46:04.779803 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 1024
I20260927 12:46:04.779886 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:46:04.780117 392 external_mini_cluster.cc:1428] Setting key 0c32ff9cc70231f8df1179730881e891
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Sep 27 12:46:04 dist-test-slave-lkn4 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513164, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:46:04.954449 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:04.780832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57119 (local address 127.0.98.62:36205)
0927 12:46:04.780986 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:04.780991 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:04.781999 (+ 1008us) server_negotiation.cc:407] Connection header received
0927 12:46:04.783708 (+ 1709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:04.783712 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:04.783765 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:04.783840 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:04.785312 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:04.785917 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:04.787609 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:04.787843 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:04.787930 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:04.950490 (+162560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:04.950495 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:04.950497 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:04.950533 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:04.952059 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:04.952706 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:04.952710 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:04.952712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:04.952811 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:04.953208 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:04.953212 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:04.953214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:04.953437 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:04.953547 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:04.954136 (+ 589us) server_negotiation.cc:299] Negotiation successful
0927 12:46:04.954269 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 12:46:04.955322 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:04.781246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36205 (local address 127.0.98.1:57119)
0927 12:46:04.781824 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:04.781883 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:04.783500 (+ 1617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:04.784030 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:04.784038 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:04.784500 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:04.785153 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:04.785173 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:04.786078 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:04.786082 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:04.787494 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:04.787506 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:04.788002 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:04.788007 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:04.788213 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:04.788885 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:04.788917 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:04.950244 (+161327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:04.952196 (+ 1952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:04.952205 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:04.952221 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:04.952576 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:04.952898 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:04.952901 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:04.952903 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:04.953103 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:04.953548 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:04.953556 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:04.953870 (+ 314us) client_negotiation.cc:769] Sending connection context
0927 12:46:04.954067 (+ 197us) client_negotiation.cc:240] Negotiation successful
0927 12:46:04.954429 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":134,"threads_started":1}
I20260927 12:46:04.956491 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36205
I20260927 12:46:04.956758 1153 heartbeater.cc:464] Registering TS with master...
I20260927 12:46:04.957273 1153 heartbeater.cc:511] Master 127.0.98.62:36205 requested a full tablet report, sending...
I20260927 12:46:04.958829 962 ts_manager.cc:194] Registered new tserver with Master: 1ff70c61dd8d402784b6ca7d80f45194 (127.0.98.1:45295)
I20260927 12:46:04.960016 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57119
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab.
W20260927 12:46:05.619298 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.2: real 0.837s user 0.000s sys 0.003s
I20260927 12:46:05.619846 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36205
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:46:05.734925 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:46:05.735219 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:46:05.735283 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:46:05.739495 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:46:05.739567 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:46:05.739652 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:46:05.745050 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36205
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:46:05.746151 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:46:05.747167 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:46:05.757088 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:05.757088 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:05.757157 1158 server_base.cc:1034] running on GCE node
W20260927 12:46:05.757124 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:05.757874 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:46:05.758558 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:46:05.759764 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790513165759724 us; error 62 us; skew 500 ppm
Sep 27 12:46:05 dist-test-slave-lkn4 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:46:05.762807 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 12:46:05.763835 1158 webserver.cc:460] Webserver started at http://127.0.98.2:38127/ using document root <none> and password file <none>
I20260927 12:46:05.764389 1158 fs_manager.cc:359] Metadata directory not provided
I20260927 12:46:05.764458 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:46:05.764662 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:46:05.766860 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/minicluster-data/ts-1/data/instance:
uuid: "bfad8bee04fa47d395cd70a190e67a41"
format_stamp: "Formatted at 2026-09-27 12:46:05 on dist-test-slave-lkn4"
server_key: "82b919817ac1ff4a657826ab0e5a3522"
server_key_iv: "ab998df8d3c63ba11a939b00dccb8bc9"
server_key_version: "encryptionkey@0"
I20260927 12:46:05.767390 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/minicluster-data/ts-1/wal/instance:
uuid: "bfad8bee04fa47d395cd70a190e67a41"
format_stamp: "Formatted at 2026-09-27 12:46:05 on dist-test-slave-lkn4"
server_key: "82b919817ac1ff4a657826ab0e5a3522"
server_key_iv: "ab998df8d3c63ba11a939b00dccb8bc9"
server_key_version: "encryptionkey@0"
I20260927 12:46:05.771376 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 12:46:05.774197 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:05.775650 1158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:46:05.775840 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/minicluster-data/ts-1/wal
uuid: "bfad8bee04fa47d395cd70a190e67a41"
format_stamp: "Formatted at 2026-09-27 12:46:05 on dist-test-slave-lkn4"
server_key: "82b919817ac1ff4a657826ab0e5a3522"
server_key_iv: "ab998df8d3c63ba11a939b00dccb8bc9"
server_key_version: "encryptionkey@0"
I20260927 12:46:05.775951 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:46:05.802328 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:46:05.805778 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:46:05.805964 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:46:05.806631 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:46:05.807495 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:46:05.807544 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:05.807598 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:46:05.807626 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:05.816721 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39263
I20260927 12:46:05.816790 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39263 every 8 connection(s)
I20260927 12:46:05.817672 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:46:05.826670 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 1158
I20260927 12:46:05.826802 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:46:05.827090 392 external_mini_cluster.cc:1428] Setting key a89333ab50ebd5604f520c8124701f08
Sep 27 12:46:05 dist-test-slave-lkn4 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:46:05.835192 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:05.819920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46241 (local address 127.0.98.62:36205)
0927 12:46:05.820810 (+ 890us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:05.820815 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:05.821197 (+ 382us) server_negotiation.cc:407] Connection header received
0927 12:46:05.822677 (+ 1480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:05.822681 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:05.822743 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:05.822832 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:05.824205 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:05.824948 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:05.826867 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:05.827257 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:05.827369 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:05.830479 (+ 3110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:05.830483 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:05.830486 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:05.830514 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:05.832411 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:05.833085 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:05.833089 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:05.833091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:05.833206 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:05.833645 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:05.833661 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:05.833663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:05.833913 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:05.834006 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:05.834837 (+ 831us) server_negotiation.cc:299] Negotiation successful
0927 12:46:05.834987 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":802,"thread_start_us":162,"threads_started":1}
I20260927 12:46:05.837033 1290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:05.820410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36205 (local address 127.0.98.2:46241)
0927 12:46:05.820986 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:05.821033 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:05.822459 (+ 1426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:05.823031 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:05.823039 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:05.823429 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:05.824051 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:05.824074 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:05.825097 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:05.825100 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:05.826639 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:05.826654 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:05.827464 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:05.827470 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:05.827824 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:05.828586 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:05.828610 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:05.830288 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:05.832542 (+ 2254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:05.832550 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:05.832571 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:05.832960 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:05.833310 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:05.833313 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:05.833315 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:05.833524 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:05.834106 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:05.834115 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:05.834505 (+ 390us) client_negotiation.cc:769] Sending connection context
0927 12:46:05.835465 (+ 960us) client_negotiation.cc:240] Negotiation successful
0927 12:46:05.835889 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":139,"threads_started":1}
I20260927 12:46:05.838507 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36205
I20260927 12:46:05.838776 1287 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 12:46:05.839424 1287 heartbeater.cc:511] Master 127.0.98.62:36205 requested a full tablet report, sending...
I20260927 12:46:05.840713 962 ts_manager.cc:194] Registered new tserver with Master: bfad8bee04fa47d395cd70a190e67a41 (127.0.98.2:39263)
I20260927 12:46:05.841477 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46241
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 12:46:05.962626 1153 heartbeater.cc:503] Master 127.0.98.62:36205 was elected leader, sending a full tablet report...
I20260927 12:46:06.843822 1287 heartbeater.cc:503] Master 127.0.98.62:36205 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab.
W20260927 12:46:06.883881 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.3: real 1.054s user 0.000s sys 0.004s
I20260927 12:46:06.884466 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36205
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:46:06.999891 1293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:46:07.000176 1293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:46:07.000236 1293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:46:07.004328 1293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:46:07.004401 1293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:46:07.004479 1293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:46:07.010197 1293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36205
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:46:07.011337 1293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:46:07.012712 1293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:46:07.022593 1301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:07.022709 1293 server_base.cc:1034] running on GCE node
W20260927 12:46:07.022544 1299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:07.022543 1298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:07.023344 1293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:46:07.024057 1293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:46:07.025224 1293 hybrid_clock.cc:648] HybridClock initialized: now 1790513167025213 us; error 25 us; skew 500 ppm
Sep 27 12:46:07 dist-test-slave-lkn4 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513167, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:46:07.035284 1293 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 12:46:07.036376 1293 webserver.cc:460] Webserver started at http://127.0.98.3:37715/ using document root <none> and password file <none>
I20260927 12:46:07.036998 1293 fs_manager.cc:359] Metadata directory not provided
I20260927 12:46:07.037063 1293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:46:07.037261 1293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:46:07.039463 1293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/minicluster-data/ts-2/data/instance:
uuid: "da7ff97df4b74bc9b840c7194d89fce5"
format_stamp: "Formatted at 2026-09-27 12:46:07 on dist-test-slave-lkn4"
server_key: "e8f9a024c9217204209ff2f2453f7848"
server_key_iv: "9e73a463a6c82b8426f7c88a57071de5"
server_key_version: "encryptionkey@0"
I20260927 12:46:07.040023 1293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/minicluster-data/ts-2/wal/instance:
uuid: "da7ff97df4b74bc9b840c7194d89fce5"
format_stamp: "Formatted at 2026-09-27 12:46:07 on dist-test-slave-lkn4"
server_key: "e8f9a024c9217204209ff2f2453f7848"
server_key_iv: "9e73a463a6c82b8426f7c88a57071de5"
server_key_version: "encryptionkey@0"
I20260927 12:46:07.043999 1293 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 12:46:07.046584 1308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:07.047972 1293 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:46:07.048086 1293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/minicluster-data/ts-2/wal
uuid: "da7ff97df4b74bc9b840c7194d89fce5"
format_stamp: "Formatted at 2026-09-27 12:46:07 on dist-test-slave-lkn4"
server_key: "e8f9a024c9217204209ff2f2453f7848"
server_key_iv: "9e73a463a6c82b8426f7c88a57071de5"
server_key_version: "encryptionkey@0"
I20260927 12:46:07.048198 1293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:46:07.065950 1293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:46:07.069458 1293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:46:07.069650 1293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:46:07.070344 1293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:46:07.071218 1293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:46:07.071261 1293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:07.071303 1293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:46:07.071331 1293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:07.080447 1293 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46381
I20260927 12:46:07.080590 1421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46381 every 8 connection(s)
I20260927 12:46:07.081439 1293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:46:07.081807 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 1293
I20260927 12:46:07.081933 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790513161160093-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:46:07.082259 392 external_mini_cluster.cc:1428] Setting key c2d38a0ee30b582e0ab5d8d86f155262
Sep 27 12:46:07 dist-test-slave-lkn4 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513167, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:46:07.095630 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:07.083512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42943 (local address 127.0.98.62:36205)
0927 12:46:07.085167 (+ 1655us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:07.085172 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:07.085186 (+ 14us) server_negotiation.cc:407] Connection header received
0927 12:46:07.086276 (+ 1090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:07.086279 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:07.086335 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:07.086425 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:07.087611 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:07.088206 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:07.089351 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:07.089585 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:07.089672 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:07.092273 (+ 2601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:07.092277 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:07.092279 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:07.092304 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:07.093597 (+ 1293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:07.094095 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:07.094099 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:07.094101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:07.094185 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:07.094481 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:07.094484 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:07.094486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:07.094680 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:07.094774 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:07.095321 (+ 547us) server_negotiation.cc:299] Negotiation successful
0927 12:46:07.095451 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1568,"thread_start_us":60,"threads_started":1}
I20260927 12:46:07.096565 1425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:07.084091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36205 (local address 127.0.98.3:42943)
0927 12:46:07.084748 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:07.084805 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:07.086066 (+ 1261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:07.086602 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:07.086610 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:07.086985 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:07.087467 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:07.087481 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:07.088331 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:07.088335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:07.089233 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:07.089243 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:07.089727 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:07.089731 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:07.089891 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:07.090459 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:07.090475 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:07.092091 (+ 1616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:07.093718 (+ 1627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:07.093724 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:07.093741 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:07.093989 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:07.094264 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:07.094267 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:07.094269 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:07.094399 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:07.094783 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:07.094790 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:07.095099 (+ 309us) client_negotiation.cc:769] Sending connection context
0927 12:46:07.095288 (+ 189us) client_negotiation.cc:240] Negotiation successful
0927 12:46:07.095619 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":137,"threads_started":1}
I20260927 12:46:07.097564 1422 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36205
I20260927 12:46:07.097877 1422 heartbeater.cc:464] Registering TS with master...
I20260927 12:46:07.098392 1422 heartbeater.cc:511] Master 127.0.98.62:36205 requested a full tablet report, sending...
I20260927 12:46:07.099329 962 ts_manager.cc:194] Registered new tserver with Master: da7ff97df4b74bc9b840c7194d89fce5 (127.0.98.3:46381)
I20260927 12:46:07.099916 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42943
I20260927 12:46:07.108246 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 12:46:07.119375 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:07.111893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39720 (local address 127.0.98.62:36205)
0927 12:46:07.112017 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:07.112022 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:07.112227 (+ 205us) server_negotiation.cc:407] Connection header received
0927 12:46:07.112371 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:07.112373 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:07.112415 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:07.112483 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:07.113322 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:07.113846 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:07.114818 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:07.115038 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:07.115115 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:07.116483 (+ 1368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:07.116486 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:07.116488 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:07.116513 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:07.117808 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:07.118242 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:07.118245 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:07.118247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:07.118356 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:07.118602 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:07.118604 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:07.118605 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:07.118807 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:07.118906 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:07.119138 (+ 232us) server_negotiation.cc:299] Negotiation successful
0927 12:46:07.119241 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 12:46:07.124989 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39720:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:46:07.128141 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:46:07.142546 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:07.136994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54706 (local address 127.0.98.3:46381)
0927 12:46:07.137445 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:07.137457 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:07.137484 (+ 27us) server_negotiation.cc:407] Connection header received
0927 12:46:07.137568 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:07.137575 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:07.137821 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:07.138369 (+ 548us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:46:07.138535 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:07.139542 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:07.140972 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:07.141831 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:07.141938 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:07.142148 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:07.142280 (+ 132us) server_negotiation.cc:299] Negotiation successful
0927 12:46:07.142392 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":147,"threads_started":1}
I20260927 12:46:07.142534 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:07.136803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46381 (local address 127.0.0.1:54706)
0927 12:46:07.137259 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:07.137317 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:07.137442 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:07.137982 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:07.137988 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:07.138013 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:46:07.138194 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:07.138202 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:07.139659 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:07.139663 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:07.140816 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:07.140848 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:07.141931 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:07.141934 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:07.142076 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:07.142125 (+ 49us) client_negotiation.cc:769] Sending connection context
0927 12:46:07.142245 (+ 120us) client_negotiation.cc:240] Negotiation successful
0927 12:46:07.142376 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":132,"threads_started":1}
I20260927 12:46:07.143242 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:07.137689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45295 (local address 127.0.0.1:33760)
0927 12:46:07.138106 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:07.138119 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:07.138201 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:07.138934 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:07.138937 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:07.138957 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:46:07.139064 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:07.139070 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:07.140622 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:07.140626 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:07.141769 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:07.141780 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:07.142833 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:07.142837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:07.142940 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:07.142965 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 12:46:07.143040 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 12:46:07.143113 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":83,"threads_started":1}
I20260927 12:46:07.143529 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:07.138070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33760 (local address 127.0.98.1:45295)
0927 12:46:07.138446 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:07.138453 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:07.138476 (+ 23us) server_negotiation.cc:407] Connection header received
0927 12:46:07.138551 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:07.138556 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:07.138754 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:07.138946 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:46:07.139180 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:07.140485 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:07.141867 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:07.142711 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:07.142820 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:07.142951 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:07.143325 (+ 374us) server_negotiation.cc:299] Negotiation successful
0927 12:46:07.143427 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":78,"threads_started":1}
I20260927 12:46:07.143530 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:07.137134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39263 (local address 127.0.0.1:41188)
0927 12:46:07.137667 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:07.137679 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:07.137753 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:07.139002 (+ 1249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:07.139005 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:07.139022 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:46:07.139131 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:07.139137 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:07.140762 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:07.140766 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:07.141955 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:07.141965 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:07.143126 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:07.143130 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:07.143230 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:07.143252 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 12:46:07.143331 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 12:46:07.143406 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":100,"threads_started":1}
I20260927 12:46:07.143972 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:07.137699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41188 (local address 127.0.98.2:39263)
0927 12:46:07.138500 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:07.138505 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:07.138523 (+ 18us) server_negotiation.cc:407] Connection header received
0927 12:46:07.138582 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:07.138586 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:07.138868 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:07.139278 (+ 410us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:46:07.139409 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:07.140654 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:07.142109 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:07.142980 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:07.143104 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:07.143229 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:07.143718 (+ 489us) server_negotiation.cc:299] Negotiation successful
0927 12:46:07.143842 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":712,"thread_start_us":84,"threads_started":1}
I20260927 12:46:07.146442 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 1ebf09c8c6b14ad4ab9b75e61a66ee49 (DEFAULT_TABLE table=test-table [id=949d637b6a9d41d4a000c80a3b9cbf34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:46:07.146476 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 1ebf09c8c6b14ad4ab9b75e61a66ee49 (DEFAULT_TABLE table=test-table [id=949d637b6a9d41d4a000c80a3b9cbf34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:46:07.146461 1356 tablet_service.cc:1467] Processing CreateTablet for tablet 1ebf09c8c6b14ad4ab9b75e61a66ee49 (DEFAULT_TABLE table=test-table [id=949d637b6a9d41d4a000c80a3b9cbf34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:46:07.147432 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ebf09c8c6b14ad4ab9b75e61a66ee49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:46:07.147451 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ebf09c8c6b14ad4ab9b75e61a66ee49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:46:07.147454 1356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ebf09c8c6b14ad4ab9b75e61a66ee49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:46:07.155114 1442 tablet_bootstrap.cc:492] T 1ebf09c8c6b14ad4ab9b75e61a66ee49 P 1ff70c61dd8d402784b6ca7d80f45194: Bootstrap starting.
I20260927 12:46:07.157177 1442 tablet_bootstrap.cc:654] T 1ebf09c8c6b14ad4ab9b75e61a66ee49 P 1ff70c61dd8d402784b6ca7d80f45194: Neither blocks nor log segments found. Creating new log.
I20260927 12:46:07.157662 1441 tablet_bootstrap.cc:492] T 1ebf09c8c6b14ad4ab9b75e61a66ee49 P bfad8bee04fa47d395cd70a190e67a41: Bootstrap starting.
I20260927 12:46:07.157953 1442 log.cc:824] T 1ebf09c8c6b14ad4ab9b75e61a66ee49 P 1ff70c61dd8d402784b6ca7d80f45194: Log is configured to *not* fsync() on all Append() calls
I20260927 12:46:07.158219 1443 tablet_bootstrap.cc:492] T 1ebf09c8c6b14ad4ab9b75e61a66ee49 P da7ff97df4b74bc9b840c7194d89fce5: Bootstrap starting.
I20260927 12:46:07.160316 1443 tablet_bootstrap.cc:654] T 1ebf09c8c6b14ad4ab9b75e61a66ee49 P da7ff97df4b74bc9b840c7194d89fce5: Neither blocks nor log segments found. Creating new log.
I20260927 12:46:07.160338 1441 tablet_bootstrap.cc:654] T 1ebf09c8c6b14ad4ab9b75e61a66ee49 P bfad8bee04fa47d395cd70a190e67a41: Neither blocks nor log segments found. Creating new log.
I20260927 12:46:07.160400 1442 tablet_bootstrap.cc:492] T 1ebf09c8c6b14ad4ab9b75e61a66ee49 P 1ff70c61dd8d402784b6ca7d80f45194: No bootstrap required, opened a new log
I20260927 12:46:07.160609 1442 ts_tablet_manager.cc:1397] T 1ebf09c8c6b14ad4ab9b75e61a66ee49 P 1ff70c61dd8d402784b6ca7d80f45194: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260927 12:46:07.161190 1441 log.cc:824] T 1ebf09c8c6b14ad4ab9b75e61a66ee49 P bfad8bee04fa47d395cd70a190e67a41: Log is configured to *not* fsync() on all Append() calls
I20260927 12:46:07.161417 1443 log.cc:824] T 1ebf09c8c6b14ad4ab9b75e61a66ee49 P da7ff97df4b74bc9b840c7194d89fce5: Log is configured to *not* fsync() on all Append() calls
I20260927 12:46:07.163178 1441 tablet_bootstrap.cc:492] T 1ebf09c8c6b14ad4ab9b75e61a66ee49 P bfad8bee04fa47d395cd70a190e67a41: No bootstrap required, opened a new log
I20260927 12:46:07.163369 1441 ts_tablet_manager.cc:1397] T 1ebf09c8c6b14ad4ab9b75e61a66ee49 P bfad8bee04fa47d395cd70a190e67a41: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 12:46:07.163393 1443 tablet_bootstrap.cc:492] T 1ebf09c8c6b14ad4ab9b75e61a66ee49 P da7ff97df4b74bc9b840c7194d89fce5: No bootstrap required, opened a new log
I20260927 12:46:07.163578 1443 ts_tablet_manager.cc:1397] T 1ebf09c8c6b14ad4ab9b75e61a66ee49 P da7ff97df4b74bc9b840c7194d89fce5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 12:46:07.165145 1442 raft_consensus.cc:348] T 1ebf09c8c6b14ad4ab9b75e61a66ee49 P 1ff70c61dd8d402784b6ca7d80f45194 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da7ff97df4b74bc9b840c7194d89fce5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46381 } } peers { permanent_uuid: "1ff70c61dd8d402784b6ca7d80f45194" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45295 } } peers { permanent_uuid: "bfad8bee04fa47d395cd70a190e67a41" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39263 } }
I20260927 12:46:07.165506 1442 raft_consensus.cc:374] T 1ebf09c8c6b14ad4ab9b75e61a66ee49 P 1ff70c61dd8d402784b6ca7d80f45194 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:46:07.165580 1442 raft_consensus.cc:729] T 1ebf09c8c6b14ad4ab9b75e61a66ee49 P 1ff70c61dd8d402784b6ca7d80f45194 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ff70c61dd8d402784b6ca7d80f45194, State: Initialized, Role: FOLLOWER
I20260927 12:46:07.166123 1442 consensus_queue.cc:260] T 1ebf09c8c6b14ad4ab9b75e61a66ee49 P 1ff70c61dd8d402784b6ca7d80f45194 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da7ff97df4b74bc9b840c7194d89fce5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46381 } } peers { permanent_uuid: "1ff70c61dd8d402784b6ca7d80f45194" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45295 } } peers { permanent_uuid: "bfad8bee04fa47d395cd70a190e67a41" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39263 } }
I20260927 12:46:07.167167 1442 ts_tablet_manager.cc:1428] T 1ebf09c8c6b14ad4ab9b75e61a66ee49 P 1ff70c61dd8d402784b6ca7d80f45194: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 12:46:07.167312 1441 raft_consensus.cc:348] T 1ebf09c8c6b14ad4ab9b75e61a66ee49 P bfad8bee04fa47d395cd70a190e67a41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da7ff97df4b74bc9b840c7194d89fce5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46381 } } peers { permanent_uuid: "1ff70c61dd8d402784b6ca7d80f45194" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45295 } } peers { permanent_uuid: "bfad8bee04fa47d395cd70a190e67a41" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39263 } }
I20260927 12:46:07.167578 1441 raft_consensus.cc:374] T 1ebf09c8c6b14ad4ab9b75e61a66ee49 P bfad8bee04fa47d395cd70a190e67a41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:46:07.167647 1441 raft_consensus.cc:729] T 1ebf09c8c6b14ad4ab9b75e61a66ee49 P bfad8bee04fa47d395cd70a190e67a41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfad8bee04fa47d395cd70a190e67a41, State: Initialized, Role: FOLLOWER
I20260927 12:46:07.167717 1443 raft_consensus.cc:348] T 1ebf09c8c6b14ad4ab9b75e61a66ee49 P da7ff97df4b74bc9b840c7194d89fce5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da7ff97df4b74bc9b840c7194d89fce5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46381 } } peers { permanent_uuid: "1ff70c61dd8d402784b6ca7d80f45194" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45295 } } peers { permanent_uuid: "bfad8bee04fa47d395cd70a190e67a41" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39263 } }
I20260927 12:46:07.167984 1443 raft_consensus.cc:374] T 1ebf09c8c6b14ad4ab9b75e61a66ee49 P da7ff97df4b74bc9b840c7194d89fce5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:46:07.168040 1443 raft_consensus.cc:729] T 1ebf09c8c6b14ad4ab9b75e61a66ee49 P da7ff97df4b74bc9b840c7194d89fce5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da7ff97df4b74bc9b840c7194d89fce5, State: Initialized, Role: FOLLOWER
I20260927 12:46:07.168157 1441 consensus_queue.cc:260] T 1ebf09c8c6b14ad4ab9b75e61a66ee49 P bfad8bee04fa47d395cd70a190e67a41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da7ff97df4b74bc9b840c7194d89fce5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46381 } } peers { permanent_uuid: "1ff70c61dd8d402784b6ca7d80f45194" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45295 } } peers { permanent_uuid: "bfad8bee04fa47d395cd70a190e67a41" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39263 } }
I20260927 12:46:07.168428 1443 consensus_queue.cc:260] T 1ebf09c8c6b14ad4ab9b75e61a66ee49 P da7ff97df4b74bc9b840c7194d89fce5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da7ff97df4b74bc9b840c7194d89fce5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46381 } } peers { permanent_uuid: "1ff70c61dd8d402784b6ca7d80f45194" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45295 } } peers { permanent_uuid: "bfad8bee04fa47d395cd70a190e67a41" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39263 } }
I20260927 12:46:07.169236 1441 ts_tablet_manager.cc:1428] T 1ebf09c8c6b14ad4ab9b75e61a66ee49 P bfad8bee04fa47d395cd70a190e67a41: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260927 12:46:07.169341 1443 ts_tablet_manager.cc:1428] T 1ebf09c8c6b14ad4ab9b75e61a66ee49 P da7ff97df4b74bc9b840c7194d89fce5: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 12:46:07.169644 1422 heartbeater.cc:503] Master 127.0.98.62:36205 was elected leader, sending a full tablet report...
I20260927 12:46:07.250550 1449 raft_consensus.cc:482] T 1ebf09c8c6b14ad4ab9b75e61a66ee49 P da7ff97df4b74bc9b840c7194d89fce5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:46:07.250869 1449 raft_consensus.cc:504] T 1ebf09c8c6b14ad4ab9b75e61a66ee49 P da7ff97df4b74bc9b840c7194d89fce5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da7ff97df4b74bc9b840c7194d89fce5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46381 } } peers { permanent_uuid: "1ff70c61dd8d402784b6ca7d80f45194" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45295 } } peers { permanent_uuid: "bfad8bee04fa47d395cd70a190e67a41" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39263 } }
I20260927 12:46:07.252082 1449 leader_election.cc:288] T 1ebf09c8c6b14ad4ab9b75e61a66ee49 P da7ff97df4b74bc9b840c7194d89fce5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ff70c61dd8d402784b6ca7d80f45194 (127.0.98.1:45295), bfad8bee04fa47d395cd70a190e67a41 (127.0.98.2:39263)
I20260927 12:46:07.256922 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:07.252525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47215 (local address 127.0.98.2:39263)
0927 12:46:07.252660 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:07.252665 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:07.253010 (+ 345us) server_negotiation.cc:407] Connection header received
0927 12:46:07.253112 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:07.253115 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:07.253153 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:07.253206 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:46:07.253462 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:07.254219 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:07.255715 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:07.256333 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:07.256441 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:07.256513 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:07.256715 (+ 202us) server_negotiation.cc:299] Negotiation successful
0927 12:46:07.256792 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 12:46:07.256925 1425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:07.252423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45295 (local address 127.0.98.3:33733)
0927 12:46:07.252615 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:07.252632 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:07.252761 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:07.253017 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:07.253020 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:07.253033 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:46:07.253148 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:07.253154 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:07.254215 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:07.254218 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:07.255566 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:07.255577 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:07.256522 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:07.256525 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:07.256614 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:07.256631 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 12:46:07.256756 (+ 125us) client_negotiation.cc:240] Negotiation successful
0927 12:46:07.256811 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":48}
I20260927 12:46:07.256989 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:07.252662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33733 (local address 127.0.98.1:45295)
0927 12:46:07.252809 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:07.252812 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:07.252825 (+ 13us) server_negotiation.cc:407] Connection header received
0927 12:46:07.252877 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:07.252879 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:07.252924 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:07.252994 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:46:07.253240 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:07.254096 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:07.255720 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:07.256418 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:07.256508 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:07.256564 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:07.256776 (+ 212us) server_negotiation.cc:299] Negotiation successful
0927 12:46:07.256887 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 12:46:07.256932 1450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:07.252423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39263 (local address 127.0.98.3:47215)
0927 12:46:07.252909 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:07.252922 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:07.253017 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:07.253247 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:07.253251 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:07.253267 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:46:07.253373 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:07.253377 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:07.254303 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:07.254306 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:07.255567 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:07.255577 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:07.256475 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:07.256483 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:07.256587 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:07.256614 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 12:46:07.256694 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 12:46:07.256767 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"mutex_wait_us":108,"thread_start_us":128,"threads_started":1}
I20260927 12:46:07.257686 1238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ebf09c8c6b14ad4ab9b75e61a66ee49" candidate_uuid: "da7ff97df4b74bc9b840c7194d89fce5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfad8bee04fa47d395cd70a190e67a41" is_pre_election: true
I20260927 12:46:07.257695 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ebf09c8c6b14ad4ab9b75e61a66ee49" candidate_uuid: "da7ff97df4b74bc9b840c7194d89fce5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ff70c61dd8d402784b6ca7d80f45194" is_pre_election: true
I20260927 12:46:07.257972 1238 raft_consensus.cc:2454] T 1ebf09c8c6b14ad4ab9b75e61a66ee49 P bfad8bee04fa47d395cd70a190e67a41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da7ff97df4b74bc9b840c7194d89fce5 in term 0.
I20260927 12:46:07.257972 1107 raft_consensus.cc:2454] T 1ebf09c8c6b14ad4ab9b75e61a66ee49 P 1ff70c61dd8d402784b6ca7d80f45194 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da7ff97df4b74bc9b840c7194d89fce5 in term 0.
I20260927 12:46:07.258428 1312 leader_election.cc:302] T 1ebf09c8c6b14ad4ab9b75e61a66ee49 P da7ff97df4b74bc9b840c7194d89fce5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bfad8bee04fa47d395cd70a190e67a41, da7ff97df4b74bc9b840c7194d89fce5; no voters:
I20260927 12:46:07.258714 1449 raft_consensus.cc:2781] T 1ebf09c8c6b14ad4ab9b75e61a66ee49 P da7ff97df4b74bc9b840c7194d89fce5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:46:07.258787 1449 raft_consensus.cc:482] T 1ebf09c8c6b14ad4ab9b75e61a66ee49 P da7ff97df4b74bc9b840c7194d89fce5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:46:07.258818 1449 raft_consensus.cc:3037] T 1ebf09c8c6b14ad4ab9b75e61a66ee49 P da7ff97df4b74bc9b840c7194d89fce5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:46:07.268936 1449 raft_consensus.cc:504] T 1ebf09c8c6b14ad4ab9b75e61a66ee49 P da7ff97df4b74bc9b840c7194d89fce5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da7ff97df4b74bc9b840c7194d89fce5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46381 } } peers { permanent_uuid: "1ff70c61dd8d402784b6ca7d80f45194" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45295 } } peers { permanent_uuid: "bfad8bee04fa47d395cd70a190e67a41" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39263 } }
I20260927 12:46:07.269457 1449 leader_election.cc:288] T 1ebf09c8c6b14ad4ab9b75e61a66ee49 P da7ff97df4b74bc9b840c7194d89fce5 [CANDIDATE]: Term 1 election: Requested vote from peers 1ff70c61dd8d402784b6ca7d80f45194 (127.0.98.1:45295), bfad8bee04fa47d395cd70a190e67a41 (127.0.98.2:39263)
I20260927 12:46:07.270017 1238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ebf09c8c6b14ad4ab9b75e61a66ee49" candidate_uuid: "da7ff97df4b74bc9b840c7194d89fce5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfad8bee04fa47d395cd70a190e67a41"
I20260927 12:46:07.270025 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ebf09c8c6b14ad4ab9b75e61a66ee49" candidate_uuid: "da7ff97df4b74bc9b840c7194d89fce5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ff70c61dd8d402784b6ca7d80f45194"
I20260927 12:46:07.270192 1238 raft_consensus.cc:3037] T 1ebf09c8c6b14ad4ab9b75e61a66ee49 P bfad8bee04fa47d395cd70a190e67a41 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:46:07.270239 1107 raft_consensus.cc:3037] T 1ebf09c8c6b14ad4ab9b75e61a66ee49 P 1ff70c61dd8d402784b6ca7d80f45194 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:46:07.271360 1107 raft_consensus.cc:2454] T 1ebf09c8c6b14ad4ab9b75e61a66ee49 P 1ff70c61dd8d402784b6ca7d80f45194 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da7ff97df4b74bc9b840c7194d89fce5 in term 1.
I20260927 12:46:07.271520 1238 raft_consensus.cc:2454] T 1ebf09c8c6b14ad4ab9b75e61a66ee49 P bfad8bee04fa47d395cd70a190e67a41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da7ff97df4b74bc9b840c7194d89fce5 in term 1.
I20260927 12:46:07.271786 1310 leader_election.cc:302] T 1ebf09c8c6b14ad4ab9b75e61a66ee49 P da7ff97df4b74bc9b840c7194d89fce5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ff70c61dd8d402784b6ca7d80f45194, da7ff97df4b74bc9b840c7194d89fce5; no voters:
I20260927 12:46:07.271970 1449 raft_consensus.cc:2781] T 1ebf09c8c6b14ad4ab9b75e61a66ee49 P da7ff97df4b74bc9b840c7194d89fce5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:46:07.272195 1449 raft_consensus.cc:686] T 1ebf09c8c6b14ad4ab9b75e61a66ee49 P da7ff97df4b74bc9b840c7194d89fce5 [term 1 LEADER]: Becoming Leader. State: Replica: da7ff97df4b74bc9b840c7194d89fce5, State: Running, Role: LEADER
I20260927 12:46:07.272603 1449 consensus_queue.cc:237] T 1ebf09c8c6b14ad4ab9b75e61a66ee49 P da7ff97df4b74bc9b840c7194d89fce5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da7ff97df4b74bc9b840c7194d89fce5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46381 } } peers { permanent_uuid: "1ff70c61dd8d402784b6ca7d80f45194" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45295 } } peers { permanent_uuid: "bfad8bee04fa47d395cd70a190e67a41" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39263 } }
I20260927 12:46:07.276119 960 catalog_manager.cc:5495] T 1ebf09c8c6b14ad4ab9b75e61a66ee49 P da7ff97df4b74bc9b840c7194d89fce5 reported cstate change: term changed from 0 to 1, leader changed from <none> to da7ff97df4b74bc9b840c7194d89fce5 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "da7ff97df4b74bc9b840c7194d89fce5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da7ff97df4b74bc9b840c7194d89fce5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46381 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ff70c61dd8d402784b6ca7d80f45194" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bfad8bee04fa47d395cd70a190e67a41" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39263 } health_report { overall_health: UNKNOWN } } }
W20260927 12:46:07.281847 1154 tablet.cc:2369] T 1ebf09c8c6b14ad4ab9b75e61a66ee49 P 1ff70c61dd8d402784b6ca7d80f45194: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 12:46:07 dist-test-slave-lkn4 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 12:46:07.299274 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:07.290315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33772 (local address 127.0.98.1:45295)
0927 12:46:07.290464 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:07.290469 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:07.290611 (+ 142us) server_negotiation.cc:407] Connection header received
0927 12:46:07.290786 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:07.290789 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:07.290839 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:07.290907 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:07.291844 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:07.292397 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:07.293271 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:07.293496 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:07.293569 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:07.296016 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:07.296027 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:07.296041 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:07.296064 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:07.297576 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:07.298030 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:07.298058 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:07.298059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:07.298131 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:07.298428 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:07.298430 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:07.298432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:07.298659 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:07.298841 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:07.299008 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 12:46:07.299100 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Sep 27 12:46:07 dist-test-slave-lkn4 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513167, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260927 12:46:07.319635 1288 tablet.cc:2369] T 1ebf09c8c6b14ad4ab9b75e61a66ee49 P bfad8bee04fa47d395cd70a190e67a41: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 12:46:07.332736 1423 tablet.cc:2369] T 1ebf09c8c6b14ad4ab9b75e61a66ee49 P da7ff97df4b74bc9b840c7194d89fce5: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 12:46:07 dist-test-slave-lkn4 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513167, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 12:46:07.357705 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:07.349114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33774 (local address 127.0.98.1:45295)
0927 12:46:07.349266 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:07.349271 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:07.349407 (+ 136us) server_negotiation.cc:407] Connection header received
0927 12:46:07.349544 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:07.349547 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:07.349593 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:07.349734 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:07.350473 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:07.350997 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:07.351855 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:07.352116 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:07.352203 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:07.354623 (+ 2420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:07.354627 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:07.354630 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:07.354728 (+ 98us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:07.355988 (+ 1260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:07.356396 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:07.356399 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:07.356401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:07.356474 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:07.356812 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:07.356816 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:07.356818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:07.356998 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:07.357124 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:07.357436 (+ 312us) server_negotiation.cc:299] Negotiation successful
0927 12:46:07.357554 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
Sep 27 12:46:07 dist-test-slave-lkn4 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 12:46:07 dist-test-slave-lkn4 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 12:46:07.441254 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:07.432914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33784 (local address 127.0.98.1:45295)
0927 12:46:07.433064 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:07.433069 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:07.433219 (+ 150us) server_negotiation.cc:407] Connection header received
0927 12:46:07.433358 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:07.433361 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:07.433407 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:07.433476 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:07.434260 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:07.434795 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:07.435642 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:07.435857 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:07.435942 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:07.438561 (+ 2619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:07.438564 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:07.438566 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:07.438591 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:07.439745 (+ 1154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:07.440136 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:07.440141 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:07.440142 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:07.440216 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:07.440456 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:07.440460 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:07.440461 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:07.440635 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:07.440745 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:07.440988 (+ 243us) server_negotiation.cc:299] Negotiation successful
0927 12:46:07.441105 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260927 12:46:07.442607 1067 tablet_service.cc:2908] Rejecting scan request for tablet 1ebf09c8c6b14ad4ab9b75e61a66ee49: Uninitialized: clean time has not yet been initialized
I20260927 12:46:07.444585 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 1024
I20260927 12:46:07.449914 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 1158
I20260927 12:46:07.455137 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 1293
I20260927 12:46:07.459878 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 930
2026-09-27T12:46:07Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4212 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:46:08 dist-test-slave-lkn4 krb5kdc[1465](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:46:08 dist-test-slave-lkn4 krb5kdc[1465](info): set up 2 sockets
Sep 27 12:46:08 dist-test-slave-lkn4 krb5kdc[1465](info): commencing operation
krb5kdc: starting...
W20260927 12:46:09.633571 392 mini_kdc.cc:121] Time spent starting KDC: real 2.152s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260927 12:46:09.776185 392 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.142s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260927 12:46:10.041147 392 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.265s user 0.000s sys 0.001s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260927 12:46:10.207392 392 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.166s user 0.000s sys 0.002s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/krb5kdc/test-user.keytab.
W20260927 12:46:10.425112 392 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.218s user 0.000s sys 0.002s
Sep 27 12:46:10 dist-test-slave-lkn4 krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513170, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:46:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:46:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:46:10.470310 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46513
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46513
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:46:10.580647 1478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:46:10.580947 1478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:46:10.581005 1478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:46:10.585057 1478 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:46:10.585129 1478 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 12:46:10.585151 1478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:46:10.585181 1478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:46:10.585212 1478 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:46:10.585242 1478 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:46:10.590672 1478 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46513
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:46:10.591812 1478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:46:10.592846 1478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:46:10.601063 1483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:10.601121 1478 server_base.cc:1034] running on GCE node
W20260927 12:46:10.601258 1486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:10.601157 1484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:10.601923 1478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:46:10.603163 1478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:46:10.604346 1478 hybrid_clock.cc:648] HybridClock initialized: now 1790513170604320 us; error 40 us; skew 500 ppm
Sep 27 12:46:10 dist-test-slave-lkn4 krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513170, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:46:10.607105 1478 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:46:10.608036 1478 webserver.cc:460] Webserver started at http://127.0.98.62:36313/ using document root <none> and password file <none>
I20260927 12:46:10.608573 1478 fs_manager.cc:359] Metadata directory not provided
I20260927 12:46:10.608637 1478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:46:10.608824 1478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:46:10.611109 1478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/minicluster-data/master-0/data/instance:
uuid: "f8fd783ac9124faa86455d32e73f8d7a"
format_stamp: "Formatted at 2026-09-27 12:46:10 on dist-test-slave-lkn4"
server_key: "d3554de8d998190a2be71f5af62e1316"
server_key_iv: "ca020f43ad186c766bf5119b2cf5dc3a"
server_key_version: "encryptionkey@0"
I20260927 12:46:10.611654 1478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/minicluster-data/master-0/wal/instance:
uuid: "f8fd783ac9124faa86455d32e73f8d7a"
format_stamp: "Formatted at 2026-09-27 12:46:10 on dist-test-slave-lkn4"
server_key: "d3554de8d998190a2be71f5af62e1316"
server_key_iv: "ca020f43ad186c766bf5119b2cf5dc3a"
server_key_version: "encryptionkey@0"
I20260927 12:46:10.615496 1478 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 12:46:10.617952 1493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:10.619292 1478 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:46:10.619424 1478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/minicluster-data/master-0/wal
uuid: "f8fd783ac9124faa86455d32e73f8d7a"
format_stamp: "Formatted at 2026-09-27 12:46:10 on dist-test-slave-lkn4"
server_key: "d3554de8d998190a2be71f5af62e1316"
server_key_iv: "ca020f43ad186c766bf5119b2cf5dc3a"
server_key_version: "encryptionkey@0"
I20260927 12:46:10.619534 1478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:46:10.648053 1478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:46:10.651237 1478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:46:10.651417 1478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:46:10.659235 1478 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46513
I20260927 12:46:10.659263 1555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46513 every 8 connection(s)
I20260927 12:46:10.660233 1478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:46:10.663064 1556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:46:10.666416 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 1478
I20260927 12:46:10.666507 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/minicluster-data/master-0/wal/instance
I20260927 12:46:10.666723 392 external_mini_cluster.cc:1428] Setting key f97f67c2f3b2332001cd3570dc04393c
I20260927 12:46:10.669130 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8fd783ac9124faa86455d32e73f8d7a: Bootstrap starting.
I20260927 12:46:10.671437 1556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8fd783ac9124faa86455d32e73f8d7a: Neither blocks nor log segments found. Creating new log.
I20260927 12:46:10.672153 1556 log.cc:824] T 00000000000000000000000000000000 P f8fd783ac9124faa86455d32e73f8d7a: Log is configured to *not* fsync() on all Append() calls
I20260927 12:46:10.674276 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8fd783ac9124faa86455d32e73f8d7a: No bootstrap required, opened a new log
Sep 27 12:46:10 dist-test-slave-lkn4 krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513170, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:46:10.678403 1556 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8fd783ac9124faa86455d32e73f8d7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8fd783ac9124faa86455d32e73f8d7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46513 } }
I20260927 12:46:10.678728 1556 raft_consensus.cc:374] T 00000000000000000000000000000000 P f8fd783ac9124faa86455d32e73f8d7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:46:10.678810 1556 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8fd783ac9124faa86455d32e73f8d7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8fd783ac9124faa86455d32e73f8d7a, State: Initialized, Role: FOLLOWER
I20260927 12:46:10.679296 1556 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8fd783ac9124faa86455d32e73f8d7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8fd783ac9124faa86455d32e73f8d7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46513 } }
I20260927 12:46:10.679430 1556 raft_consensus.cc:388] T 00000000000000000000000000000000 P f8fd783ac9124faa86455d32e73f8d7a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:46:10.679497 1556 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8fd783ac9124faa86455d32e73f8d7a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:46:10.679618 1556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8fd783ac9124faa86455d32e73f8d7a [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:46:10.680667 1556 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8fd783ac9124faa86455d32e73f8d7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8fd783ac9124faa86455d32e73f8d7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46513 } }
I20260927 12:46:10.680995 1556 leader_election.cc:302] T 00000000000000000000000000000000 P f8fd783ac9124faa86455d32e73f8d7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f8fd783ac9124faa86455d32e73f8d7a; no voters:
I20260927 12:46:10.681069 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:10.668224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58402 (local address 127.0.98.62:46513)
0927 12:46:10.668645 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:10.668660 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:10.668695 (+ 35us) server_negotiation.cc:407] Connection header received
0927 12:46:10.669321 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:10.669341 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:10.669669 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:10.669968 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:10.670726 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:10.671738 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:10.672581 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:10.673052 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:10.673200 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:10.676049 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:10.676054 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:10.676068 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:10.676095 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:10.677907 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:10.678487 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:10.678492 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:10.678496 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:10.678612 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:10.678976 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:10.678979 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:10.678980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:10.679468 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:10.679675 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:10.679967 (+ 292us) server_negotiation.cc:299] Negotiation successful
0927 12:46:10.680269 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":132,"threads_started":1}
I20260927 12:46:10.681326 1556 leader_election.cc:288] T 00000000000000000000000000000000 P f8fd783ac9124faa86455d32e73f8d7a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:46:10.681448 1561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f8fd783ac9124faa86455d32e73f8d7a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:46:10.681649 1561 raft_consensus.cc:686] T 00000000000000000000000000000000 P f8fd783ac9124faa86455d32e73f8d7a [term 1 LEADER]: Becoming Leader. State: Replica: f8fd783ac9124faa86455d32e73f8d7a, State: Running, Role: LEADER
I20260927 12:46:10.681962 1561 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8fd783ac9124faa86455d32e73f8d7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8fd783ac9124faa86455d32e73f8d7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46513 } }
I20260927 12:46:10.682729 1556 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8fd783ac9124faa86455d32e73f8d7a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:46:10.683863 1562 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8fd783ac9124faa86455d32e73f8d7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f8fd783ac9124faa86455d32e73f8d7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8fd783ac9124faa86455d32e73f8d7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46513 } } }
I20260927 12:46:10.683988 1562 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8fd783ac9124faa86455d32e73f8d7a [sys.catalog]: This master's current role is: LEADER
I20260927 12:46:10.684276 1563 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8fd783ac9124faa86455d32e73f8d7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8fd783ac9124faa86455d32e73f8d7a. Latest consensus state: current_term: 1 leader_uuid: "f8fd783ac9124faa86455d32e73f8d7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8fd783ac9124faa86455d32e73f8d7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46513 } } }
I20260927 12:46:10.684361 1563 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8fd783ac9124faa86455d32e73f8d7a [sys.catalog]: This master's current role is: LEADER
I20260927 12:46:10.684428 1570 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:46:10.688059 1570 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:46:10.695998 1570 catalog_manager.cc:1347] Generated new cluster ID: a6b553d185db497b86a89fa8dc070111
I20260927 12:46:10.696084 1570 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:46:10.705492 1570 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:46:10.707011 1570 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:46:10.713486 1570 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f8fd783ac9124faa86455d32e73f8d7a: Generated new TSK 0
I20260927 12:46:10.714341 1570 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 12:46:10.747535 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46513
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:46:10.859056 1582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:46:10.859341 1582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:46:10.859386 1582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:46:10.863449 1582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:46:10.863513 1582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:46:10.863552 1582 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 12:46:10.863592 1582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:46:10.868829 1582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:46:10.869922 1582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:46:10.870980 1582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:46:10.880554 1590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:10.880577 1588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:10.880579 1587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:10.880723 1582 server_base.cc:1034] running on GCE node
I20260927 12:46:10.881196 1582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:46:10.882059 1582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:46:10.883273 1582 hybrid_clock.cc:648] HybridClock initialized: now 1790513170883250 us; error 43 us; skew 500 ppm
Sep 27 12:46:10 dist-test-slave-lkn4 krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513170, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:46:10.886265 1582 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 12:46:10.887286 1582 webserver.cc:460] Webserver started at http://127.0.98.1:46769/ using document root <none> and password file <none>
I20260927 12:46:10.887887 1582 fs_manager.cc:359] Metadata directory not provided
I20260927 12:46:10.887959 1582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:46:10.888163 1582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:46:10.890313 1582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/minicluster-data/ts-0/data/instance:
uuid: "4fff3a5975fc41b488a24500cb8a3e42"
format_stamp: "Formatted at 2026-09-27 12:46:10 on dist-test-slave-lkn4"
server_key: "5b49932dfd98430d2213fbf7c3a23fd9"
server_key_iv: "240dce6795303466f493dee96d5f8bfc"
server_key_version: "encryptionkey@0"
I20260927 12:46:10.890877 1582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4fff3a5975fc41b488a24500cb8a3e42"
format_stamp: "Formatted at 2026-09-27 12:46:10 on dist-test-slave-lkn4"
server_key: "5b49932dfd98430d2213fbf7c3a23fd9"
server_key_iv: "240dce6795303466f493dee96d5f8bfc"
server_key_version: "encryptionkey@0"
I20260927 12:46:10.894547 1582 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 12:46:10.897010 1597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:10.898262 1582 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:46:10.898367 1582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/minicluster-data/ts-0/wal
uuid: "4fff3a5975fc41b488a24500cb8a3e42"
format_stamp: "Formatted at 2026-09-27 12:46:10 on dist-test-slave-lkn4"
server_key: "5b49932dfd98430d2213fbf7c3a23fd9"
server_key_iv: "240dce6795303466f493dee96d5f8bfc"
server_key_version: "encryptionkey@0"
I20260927 12:46:10.898463 1582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:46:10.919478 1582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:46:10.922850 1582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:46:10.923026 1582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:46:10.924659 1582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:46:10.924757 1582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:10.924894 1582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:46:10.924973 1582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 12:46:10 dist-test-slave-lkn4 krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513170, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:46:10.934791 1582 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34355
I20260927 12:46:10.934924 1712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34355 every 8 connection(s)
I20260927 12:46:10.935693 1582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:46:10.937453 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:10.925539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60705 (local address 127.0.98.62:46513)
0927 12:46:10.925687 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:10.925692 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:10.926691 (+ 999us) server_negotiation.cc:407] Connection header received
0927 12:46:10.927935 (+ 1244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:10.927938 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:10.927999 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:10.928084 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:10.929326 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:10.929848 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:10.930996 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:10.931253 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:10.931344 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:10.934005 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:10.934009 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:10.934010 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:10.934054 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:10.935468 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:10.935986 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:10.935989 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:10.935990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:10.936061 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:10.936372 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:10.936375 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:10.936377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:10.936592 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:10.936771 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:10.937218 (+ 447us) server_negotiation.cc:299] Negotiation successful
0927 12:46:10.937331 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 12:46:10.938165 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:10.925984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46513 (local address 127.0.98.1:60705)
0927 12:46:10.926508 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:10.926569 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:10.927611 (+ 1042us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:10.928290 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:10.928298 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:10.928634 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:10.929189 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:10.929205 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:10.929957 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:10.929961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:10.930873 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:10.930882 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:10.931393 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:10.931397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:10.931578 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:10.932117 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:10.932145 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:10.933850 (+ 1705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:10.935572 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:10.935577 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:10.935590 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:10.935894 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:10.936152 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:10.936155 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:10.936157 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:10.936290 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:10.936766 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:10.936771 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:10.936985 (+ 214us) client_negotiation.cc:769] Sending connection context
0927 12:46:10.937133 (+ 148us) client_negotiation.cc:240] Negotiation successful
0927 12:46:10.937371 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":123,"threads_started":1}
I20260927 12:46:10.939399 1713 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46513
I20260927 12:46:10.939530 1713 heartbeater.cc:464] Registering TS with master...
I20260927 12:46:10.940065 1713 heartbeater.cc:511] Master 127.0.98.62:46513 requested a full tablet report, sending...
I20260927 12:46:10.941649 1509 ts_manager.cc:194] Registered new tserver with Master: 4fff3a5975fc41b488a24500cb8a3e42 (127.0.98.1:34355)
I20260927 12:46:10.943048 1509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60705
I20260927 12:46:10.944073 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 1582
I20260927 12:46:10.944180 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:46:10.944437 392 external_mini_cluster.cc:1428] Setting key 7163b907d7b269270839d1dde98815f3
I20260927 12:46:10.949044 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:10.941743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55011 (local address 127.0.98.62:46513)
0927 12:46:10.941849 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:10.941853 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:10.942166 (+ 313us) server_negotiation.cc:407] Connection header received
0927 12:46:10.942244 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:10.942247 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:10.942288 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:10.942364 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:10.942727 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:10.943409 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:10.944466 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:10.944757 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:10.944853 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:10.946131 (+ 1278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:10.946136 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:10.946139 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:10.946162 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:10.947518 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:10.947911 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:10.947915 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:10.947916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:10.947991 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:10.948220 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:10.948223 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:10.948225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:10.948399 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:10.948482 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:10.948774 (+ 292us) server_negotiation.cc:299] Negotiation successful
0927 12:46:10.948885 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":32}
I20260927 12:46:10.949051 1719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:10.941691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46513 (local address 127.0.98.1:55011)
0927 12:46:10.942090 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:10.942104 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:10.942183 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:10.942374 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:10.942376 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:10.942535 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:10.942631 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:10.942636 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:10.943541 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:10.943544 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:10.944348 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:10.944360 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:10.944953 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:10.944957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:10.945053 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:10.945541 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:10.945555 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:10.945971 (+ 416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:10.947631 (+ 1660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:10.947634 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:10.947636 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:10.947824 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:10.948065 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:10.948068 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:10.948070 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:10.948146 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:10.948480 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:10.948484 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:10.948622 (+ 138us) client_negotiation.cc:769] Sending connection context
0927 12:46:10.948711 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 12:46:10.948858 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":203,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 12:46:10.969717 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46513
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:46:11.079983 1722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:46:11.080258 1722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:46:11.080305 1722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:46:11.084455 1722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:46:11.084517 1722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:46:11.084556 1722 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 12:46:11.084604 1722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:46:11.089912 1722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:46:11.091010 1722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:46:11.092041 1722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:46:11.101629 1730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:11.101655 1728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:11.101645 1727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:11.101701 1722 server_base.cc:1034] running on GCE node
I20260927 12:46:11.102334 1722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:46:11.103001 1722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:46:11.104164 1722 hybrid_clock.cc:648] HybridClock initialized: now 1790513171104132 us; error 44 us; skew 500 ppm
Sep 27 12:46:11 dist-test-slave-lkn4 krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:46:11.106886 1722 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 12:46:11.107784 1722 webserver.cc:460] Webserver started at http://127.0.98.2:39461/ using document root <none> and password file <none>
I20260927 12:46:11.108310 1722 fs_manager.cc:359] Metadata directory not provided
I20260927 12:46:11.108373 1722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:46:11.108563 1722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:46:11.110589 1722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/minicluster-data/ts-1/data/instance:
uuid: "ae350714d3e54c5697b0b8e1c83ac090"
format_stamp: "Formatted at 2026-09-27 12:46:11 on dist-test-slave-lkn4"
server_key: "1dba30175a2388eec3cf4207c1bddd15"
server_key_iv: "f43cf6daa044ffb8276857ed2ff840a7"
server_key_version: "encryptionkey@0"
I20260927 12:46:11.111168 1722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ae350714d3e54c5697b0b8e1c83ac090"
format_stamp: "Formatted at 2026-09-27 12:46:11 on dist-test-slave-lkn4"
server_key: "1dba30175a2388eec3cf4207c1bddd15"
server_key_iv: "f43cf6daa044ffb8276857ed2ff840a7"
server_key_version: "encryptionkey@0"
I20260927 12:46:11.115026 1722 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 12:46:11.117573 1737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:11.118980 1722 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:46:11.119093 1722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/minicluster-data/ts-1/wal
uuid: "ae350714d3e54c5697b0b8e1c83ac090"
format_stamp: "Formatted at 2026-09-27 12:46:11 on dist-test-slave-lkn4"
server_key: "1dba30175a2388eec3cf4207c1bddd15"
server_key_iv: "f43cf6daa044ffb8276857ed2ff840a7"
server_key_version: "encryptionkey@0"
I20260927 12:46:11.119201 1722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:46:11.130920 1722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:46:11.133867 1722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:46:11.134076 1722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:46:11.135442 1722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:46:11.135488 1722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:11.135541 1722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:46:11.135571 1722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:11.144935 1722 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38285
I20260927 12:46:11.145030 1852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38285 every 8 connection(s)
Sep 27 12:46:11 dist-test-slave-lkn4 krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:46:11.145931 1722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:46:11.149107 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:11.136338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45061 (local address 127.0.98.62:46513)
0927 12:46:11.136480 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:11.136484 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:11.137338 (+ 854us) server_negotiation.cc:407] Connection header received
0927 12:46:11.138527 (+ 1189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:11.138532 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:11.138591 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:11.138683 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:11.140288 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:11.141121 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:11.142619 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:11.142944 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:11.143062 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:11.145749 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:11.145753 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:11.145756 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:11.145786 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:11.147297 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:11.147798 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:11.147800 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:11.147802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:11.147877 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:11.148174 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:11.148176 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:11.148177 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:11.148368 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:11.148439 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:11.148902 (+ 463us) server_negotiation.cc:299] Negotiation successful
0927 12:46:11.148992 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 12:46:11.149995 1746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:11.136733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46513 (local address 127.0.98.2:45061)
0927 12:46:11.137172 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:11.137223 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:11.138282 (+ 1059us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:11.138896 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:11.138904 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:11.139354 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:11.140122 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:11.140144 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:11.141243 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:11.141247 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:11.142483 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:11.142493 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:11.143123 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:11.143127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:11.143331 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:11.144008 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:11.144030 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:11.145569 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:11.147428 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:11.147433 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:11.147445 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:11.147712 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:11.147946 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:11.147948 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:11.147949 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:11.148077 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:11.148455 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:11.148459 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:11.148703 (+ 244us) client_negotiation.cc:769] Sending connection context
0927 12:46:11.148868 (+ 165us) client_negotiation.cc:240] Negotiation successful
0927 12:46:11.149162 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":108,"threads_started":1}
I20260927 12:46:11.151201 1853 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46513
I20260927 12:46:11.151407 1853 heartbeater.cc:464] Registering TS with master...
I20260927 12:46:11.151906 1853 heartbeater.cc:511] Master 127.0.98.62:46513 requested a full tablet report, sending...
I20260927 12:46:11.152948 1509 ts_manager.cc:194] Registered new tserver with Master: ae350714d3e54c5697b0b8e1c83ac090 (127.0.98.2:38285)
I20260927 12:46:11.153580 1509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45061
I20260927 12:46:11.156260 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 1722
I20260927 12:46:11.156407 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:46:11.156723 392 external_mini_cluster.cc:1428] Setting key 37901a3d7009a2c4e9e5682deb97f73f
I20260927 12:46:11.160651 1859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:11.153311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46513 (local address 127.0.98.2:38279)
0927 12:46:11.153622 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:11.153636 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:11.153773 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:11.154013 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:11.154015 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:11.154225 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:11.154322 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:11.154327 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:11.155275 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:11.155278 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:11.156046 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:11.156060 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:11.156498 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:11.156502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:11.156586 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:11.157082 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:11.157095 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:11.157471 (+ 376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:11.159130 (+ 1659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:11.159134 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:11.159136 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:11.159382 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:11.159681 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:11.159683 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:11.159684 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:11.159754 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:11.160210 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:11.160212 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:11.160283 (+ 71us) client_negotiation.cc:769] Sending connection context
0927 12:46:11.160358 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 12:46:11.160465 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":95,"threads_started":1}
I20260927 12:46:11.160715 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:11.153409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38279 (local address 127.0.98.62:46513)
0927 12:46:11.153569 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:11.153573 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:11.153707 (+ 134us) server_negotiation.cc:407] Connection header received
0927 12:46:11.153868 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:11.153871 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:11.153917 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:11.153984 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:11.154423 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:11.155131 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:11.156171 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:11.156402 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:11.156495 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:11.157585 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:11.157588 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:11.157591 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:11.157615 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:11.159023 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:11.159488 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:11.159492 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:11.159494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:11.159595 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:11.159825 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:11.159829 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:11.159830 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:11.160125 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:11.160234 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:11.160433 (+ 199us) server_negotiation.cc:299] Negotiation successful
0927 12:46:11.160551 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 12:46:11.181340 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46513
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:46:11.291949 1862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:46:11.292223 1862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:46:11.292294 1862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:46:11.296296 1862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:46:11.296365 1862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:46:11.296422 1862 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 12:46:11.296478 1862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:46:11.302016 1862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1862
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:46:11.303134 1862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:46:11.304173 1862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:46:11.313439 1870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:11.313477 1867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:11.313572 1862 server_base.cc:1034] running on GCE node
W20260927 12:46:11.313498 1868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:11.314074 1862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:46:11.314702 1862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:46:11.315860 1862 hybrid_clock.cc:648] HybridClock initialized: now 1790513171315833 us; error 48 us; skew 500 ppm
Sep 27 12:46:11 dist-test-slave-lkn4 krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:46:11.318528 1862 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 12:46:11.319348 1862 webserver.cc:460] Webserver started at http://127.0.98.3:36851/ using document root <none> and password file <none>
I20260927 12:46:11.319864 1862 fs_manager.cc:359] Metadata directory not provided
I20260927 12:46:11.319927 1862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:46:11.320102 1862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:46:11.322093 1862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/minicluster-data/ts-2/data/instance:
uuid: "58e5f2ac64854c16bef4af9c707f5074"
format_stamp: "Formatted at 2026-09-27 12:46:11 on dist-test-slave-lkn4"
server_key: "d4e1bfc296a224b3b0eb469eb50accc5"
server_key_iv: "cd578d69e55c8061057e09653afc56dc"
server_key_version: "encryptionkey@0"
I20260927 12:46:11.322682 1862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/minicluster-data/ts-2/wal/instance:
uuid: "58e5f2ac64854c16bef4af9c707f5074"
format_stamp: "Formatted at 2026-09-27 12:46:11 on dist-test-slave-lkn4"
server_key: "d4e1bfc296a224b3b0eb469eb50accc5"
server_key_iv: "cd578d69e55c8061057e09653afc56dc"
server_key_version: "encryptionkey@0"
I20260927 12:46:11.326385 1862 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:46:11.328712 1877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:11.329932 1862 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:46:11.330031 1862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/minicluster-data/ts-2/wal
uuid: "58e5f2ac64854c16bef4af9c707f5074"
format_stamp: "Formatted at 2026-09-27 12:46:11 on dist-test-slave-lkn4"
server_key: "d4e1bfc296a224b3b0eb469eb50accc5"
server_key_iv: "cd578d69e55c8061057e09653afc56dc"
server_key_version: "encryptionkey@0"
I20260927 12:46:11.330189 1862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:46:11.356323 1862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:46:11.359501 1862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:46:11.359680 1862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:46:11.361094 1862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:46:11.361143 1862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:11.361204 1862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:46:11.361229 1862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 12:46:11 dist-test-slave-lkn4 krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:46:11.370645 1862 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44099
I20260927 12:46:11.370697 1992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44099 every 8 connection(s)
I20260927 12:46:11.371639 1862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:46:11.374579 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:11.362026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42685 (local address 127.0.98.62:46513)
0927 12:46:11.362188 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:11.362192 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:11.363063 (+ 871us) server_negotiation.cc:407] Connection header received
0927 12:46:11.364173 (+ 1110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:11.364177 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:11.364238 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:11.364333 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:11.365804 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:11.366450 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:11.367795 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:11.368040 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:11.368133 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:11.371130 (+ 2997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:11.371134 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:11.371137 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:11.371167 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:11.372597 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:11.373129 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:11.373132 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:11.373134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:11.373220 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:11.373520 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:11.373523 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:11.373525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:11.373711 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:11.373796 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:11.374301 (+ 505us) server_negotiation.cc:299] Negotiation successful
0927 12:46:11.374419 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 12:46:11.375360 1886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:11.362448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46513 (local address 127.0.98.3:42685)
0927 12:46:11.362893 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:11.362943 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:11.363951 (+ 1008us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:11.364546 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:11.364556 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:11.364969 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:11.365658 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:11.365677 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:11.366578 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:11.366581 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:11.367673 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:11.367684 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:11.368193 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:11.368197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:11.368375 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:11.368996 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:11.369019 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:11.370942 (+ 1923us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:11.372711 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:11.372718 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:11.372733 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:11.373035 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:11.373298 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:11.373301 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:11.373303 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:11.373434 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:11.373805 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:11.373811 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:11.374065 (+ 254us) client_negotiation.cc:769] Sending connection context
0927 12:46:11.374249 (+ 184us) client_negotiation.cc:240] Negotiation successful
0927 12:46:11.374554 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":124,"threads_started":1}
I20260927 12:46:11.376444 1993 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46513
I20260927 12:46:11.376582 1993 heartbeater.cc:464] Registering TS with master...
I20260927 12:46:11.377130 1993 heartbeater.cc:511] Master 127.0.98.62:46513 requested a full tablet report, sending...
I20260927 12:46:11.377641 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 1862
I20260927 12:46:11.377854 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:46:11.378178 392 external_mini_cluster.cc:1428] Setting key fecb95e8bc880e999ac16cb49f20e6ef
I20260927 12:46:11.378289 1510 ts_manager.cc:194] Registered new tserver with Master: 58e5f2ac64854c16bef4af9c707f5074 (127.0.98.3:44099)
I20260927 12:46:11.379012 1510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42685
I20260927 12:46:11.380054 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 12:46:11.386380 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:11.378827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32983 (local address 127.0.98.62:46513)
0927 12:46:11.378945 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:11.378949 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:11.379165 (+ 216us) server_negotiation.cc:407] Connection header received
0927 12:46:11.379382 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:11.379385 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:11.379428 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:11.379498 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:11.379973 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:11.380693 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:11.381687 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:11.381900 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:11.381984 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:11.383236 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:11.383240 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:11.383243 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:11.383267 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:11.384856 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:11.385243 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:11.385246 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:11.385249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:11.385336 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:11.385573 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:11.385576 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:11.385578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:11.385783 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:11.385858 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:11.386150 (+ 292us) server_negotiation.cc:299] Negotiation successful
0927 12:46:11.386262 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 12:46:11.386456 1999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:11.378766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46513 (local address 127.0.98.3:32983)
0927 12:46:11.379085 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:11.379101 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:11.379203 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:11.379531 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:11.379534 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:11.379742 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:11.379880 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:11.379887 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:11.380822 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:11.380826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:11.381538 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:11.381548 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:11.381998 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:11.382001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:11.382122 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:11.382760 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:11.382775 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:11.383111 (+ 336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:11.384956 (+ 1845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:11.384958 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:11.384960 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:11.385153 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:11.385425 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:11.385428 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:11.385429 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:11.385501 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:11.385872 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:11.385876 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:11.385975 (+ 99us) client_negotiation.cc:769] Sending connection context
0927 12:46:11.386110 (+ 135us) client_negotiation.cc:240] Negotiation successful
0927 12:46:11.386261 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"mutex_wait_us":7,"thread_start_us":84,"threads_started":1}
Sep 27 12:46:11 dist-test-slave-lkn4 krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513171, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 12:46:11 dist-test-slave-lkn4 krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513171, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:46:11.405910 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:11.397975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58404 (local address 127.0.98.62:46513)
0927 12:46:11.398103 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:11.398106 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:11.398118 (+ 12us) server_negotiation.cc:407] Connection header received
0927 12:46:11.398157 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:11.398159 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:11.398206 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:11.398272 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:11.398930 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:11.399488 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:11.400343 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:11.400584 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:11.400751 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:11.403123 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:11.403126 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:11.403128 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:11.403153 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:11.404475 (+ 1322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:11.404825 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:11.404828 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:11.404830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:11.404904 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:11.405108 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:11.405111 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:11.405112 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:11.405306 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:11.405406 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:11.405573 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 12:46:11.405746 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 12:46:11.408825 1510 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58404:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:46:11.412087 1510 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:46:11.427249 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:11.420943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44099 (local address 127.0.0.1:37512)
0927 12:46:11.421971 (+ 1028us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:11.422028 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:11.422196 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:11.423052 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:11.423054 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:11.423071 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:46:11.423152 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:11.423158 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:11.424603 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:11.424606 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:11.425655 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:11.425664 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:11.426648 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:11.426683 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:11.426800 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:11.426843 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 12:46:11.426958 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 12:46:11.427085 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":844,"mutex_wait_us":65,"thread_start_us":92,"threads_started":1}
I20260927 12:46:11.427546 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:11.421757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35128 (local address 127.0.98.1:34355)
0927 12:46:11.422134 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:11.422150 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:11.422304 (+ 154us) server_negotiation.cc:407] Connection header received
0927 12:46:11.422436 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:11.422441 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:11.422722 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:11.422893 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:46:11.423227 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:11.424354 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:11.426097 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:11.426869 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:11.426964 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:11.427122 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:11.427170 (+ 48us) server_negotiation.cc:299] Negotiation successful
0927 12:46:11.427263 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":64,"threads_started":1}
I20260927 12:46:11.427573 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:11.421199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38285 (local address 127.0.0.1:38360)
0927 12:46:11.422493 (+ 1294us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:11.422503 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:11.422574 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:11.423138 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:11.423141 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:11.423157 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:46:11.423246 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:11.423252 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:11.424598 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:11.424601 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:11.426160 (+ 1559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:11.426168 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:11.427227 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:11.427231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:11.427316 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:11.427338 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 12:46:11.427404 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 12:46:11.427465 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1243,"mutex_wait_us":3,"thread_start_us":90,"threads_started":1}
I20260927 12:46:11.427390 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:11.420956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34355 (local address 127.0.0.1:35128)
0927 12:46:11.422199 (+ 1243us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:11.422212 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:11.422315 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:11.422890 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:11.422897 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:11.422919 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:46:11.423103 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:11.423111 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:11.424460 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:11.424463 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:11.425969 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:11.425979 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:11.426972 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:11.426976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:11.427057 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:11.427078 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 12:46:11.427136 (+ 58us) client_negotiation.cc:240] Negotiation successful
0927 12:46:11.427203 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1182,"mutex_wait_us":58,"thread_start_us":115,"threads_started":1}
I20260927 12:46:11.428284 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:11.421803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38360 (local address 127.0.98.2:38285)
0927 12:46:11.422213 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:11.422225 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:11.422608 (+ 383us) server_negotiation.cc:407] Connection header received
0927 12:46:11.422686 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:11.422690 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:11.422845 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:11.422988 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:46:11.423368 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:11.424487 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:11.426261 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:11.426918 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:11.427006 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:11.427171 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:11.428038 (+ 867us) server_negotiation.cc:299] Negotiation successful
0927 12:46:11.428153 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":91,"threads_started":1}
I20260927 12:46:11.428329 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:11.421066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37512 (local address 127.0.98.3:44099)
0927 12:46:11.422469 (+ 1403us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:11.422479 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:11.422499 (+ 20us) server_negotiation.cc:407] Connection header received
0927 12:46:11.422559 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:11.422563 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:11.422721 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:11.422893 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:46:11.423249 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:11.424467 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:11.425773 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:11.426520 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:11.426631 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:11.426803 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:11.428116 (+ 1313us) server_negotiation.cc:299] Negotiation successful
0927 12:46:11.428207 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1305,"thread_start_us":89,"threads_started":1}
I20260927 12:46:11.430718 1647 tablet_service.cc:1467] Processing CreateTablet for tablet ca474db52dc84da497e42e2d5a5cb628 (DEFAULT_TABLE table=test-table [id=17ae78770d2f431ca16ac10bb0dc9e97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:46:11.430922 1787 tablet_service.cc:1467] Processing CreateTablet for tablet ca474db52dc84da497e42e2d5a5cb628 (DEFAULT_TABLE table=test-table [id=17ae78770d2f431ca16ac10bb0dc9e97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:46:11.430922 1927 tablet_service.cc:1467] Processing CreateTablet for tablet ca474db52dc84da497e42e2d5a5cb628 (DEFAULT_TABLE table=test-table [id=17ae78770d2f431ca16ac10bb0dc9e97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:46:11.431793 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca474db52dc84da497e42e2d5a5cb628. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:46:11.431910 1927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca474db52dc84da497e42e2d5a5cb628. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:46:11.431905 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca474db52dc84da497e42e2d5a5cb628. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:46:11.439330 2017 tablet_bootstrap.cc:492] T ca474db52dc84da497e42e2d5a5cb628 P 58e5f2ac64854c16bef4af9c707f5074: Bootstrap starting.
I20260927 12:46:11.440675 2018 tablet_bootstrap.cc:492] T ca474db52dc84da497e42e2d5a5cb628 P ae350714d3e54c5697b0b8e1c83ac090: Bootstrap starting.
I20260927 12:46:11.441511 2017 tablet_bootstrap.cc:654] T ca474db52dc84da497e42e2d5a5cb628 P 58e5f2ac64854c16bef4af9c707f5074: Neither blocks nor log segments found. Creating new log.
I20260927 12:46:11.442365 2018 tablet_bootstrap.cc:654] T ca474db52dc84da497e42e2d5a5cb628 P ae350714d3e54c5697b0b8e1c83ac090: Neither blocks nor log segments found. Creating new log.
I20260927 12:46:11.442349 2017 log.cc:824] T ca474db52dc84da497e42e2d5a5cb628 P 58e5f2ac64854c16bef4af9c707f5074: Log is configured to *not* fsync() on all Append() calls
I20260927 12:46:11.442418 2016 tablet_bootstrap.cc:492] T ca474db52dc84da497e42e2d5a5cb628 P 4fff3a5975fc41b488a24500cb8a3e42: Bootstrap starting.
I20260927 12:46:11.443135 2018 log.cc:824] T ca474db52dc84da497e42e2d5a5cb628 P ae350714d3e54c5697b0b8e1c83ac090: Log is configured to *not* fsync() on all Append() calls
I20260927 12:46:11.444345 2016 tablet_bootstrap.cc:654] T ca474db52dc84da497e42e2d5a5cb628 P 4fff3a5975fc41b488a24500cb8a3e42: Neither blocks nor log segments found. Creating new log.
I20260927 12:46:11.444454 2017 tablet_bootstrap.cc:492] T ca474db52dc84da497e42e2d5a5cb628 P 58e5f2ac64854c16bef4af9c707f5074: No bootstrap required, opened a new log
I20260927 12:46:11.444646 2017 ts_tablet_manager.cc:1397] T ca474db52dc84da497e42e2d5a5cb628 P 58e5f2ac64854c16bef4af9c707f5074: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 12:46:11.444727 2018 tablet_bootstrap.cc:492] T ca474db52dc84da497e42e2d5a5cb628 P ae350714d3e54c5697b0b8e1c83ac090: No bootstrap required, opened a new log
I20260927 12:46:11.444886 2018 ts_tablet_manager.cc:1397] T ca474db52dc84da497e42e2d5a5cb628 P ae350714d3e54c5697b0b8e1c83ac090: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260927 12:46:11.445112 2016 log.cc:824] T ca474db52dc84da497e42e2d5a5cb628 P 4fff3a5975fc41b488a24500cb8a3e42: Log is configured to *not* fsync() on all Append() calls
I20260927 12:46:11.447700 2017 raft_consensus.cc:348] T ca474db52dc84da497e42e2d5a5cb628 P 58e5f2ac64854c16bef4af9c707f5074 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e5f2ac64854c16bef4af9c707f5074" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44099 } } peers { permanent_uuid: "4fff3a5975fc41b488a24500cb8a3e42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34355 } } peers { permanent_uuid: "ae350714d3e54c5697b0b8e1c83ac090" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38285 } }
I20260927 12:46:11.447923 2017 raft_consensus.cc:374] T ca474db52dc84da497e42e2d5a5cb628 P 58e5f2ac64854c16bef4af9c707f5074 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:46:11.447956 2017 raft_consensus.cc:729] T ca474db52dc84da497e42e2d5a5cb628 P 58e5f2ac64854c16bef4af9c707f5074 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58e5f2ac64854c16bef4af9c707f5074, State: Initialized, Role: FOLLOWER
I20260927 12:46:11.448189 2018 raft_consensus.cc:348] T ca474db52dc84da497e42e2d5a5cb628 P ae350714d3e54c5697b0b8e1c83ac090 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e5f2ac64854c16bef4af9c707f5074" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44099 } } peers { permanent_uuid: "4fff3a5975fc41b488a24500cb8a3e42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34355 } } peers { permanent_uuid: "ae350714d3e54c5697b0b8e1c83ac090" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38285 } }
I20260927 12:46:11.448370 2017 consensus_queue.cc:260] T ca474db52dc84da497e42e2d5a5cb628 P 58e5f2ac64854c16bef4af9c707f5074 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e5f2ac64854c16bef4af9c707f5074" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44099 } } peers { permanent_uuid: "4fff3a5975fc41b488a24500cb8a3e42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34355 } } peers { permanent_uuid: "ae350714d3e54c5697b0b8e1c83ac090" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38285 } }
I20260927 12:46:11.448441 2018 raft_consensus.cc:374] T ca474db52dc84da497e42e2d5a5cb628 P ae350714d3e54c5697b0b8e1c83ac090 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:46:11.448498 2018 raft_consensus.cc:729] T ca474db52dc84da497e42e2d5a5cb628 P ae350714d3e54c5697b0b8e1c83ac090 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae350714d3e54c5697b0b8e1c83ac090, State: Initialized, Role: FOLLOWER
I20260927 12:46:11.448897 2018 consensus_queue.cc:260] T ca474db52dc84da497e42e2d5a5cb628 P ae350714d3e54c5697b0b8e1c83ac090 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e5f2ac64854c16bef4af9c707f5074" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44099 } } peers { permanent_uuid: "4fff3a5975fc41b488a24500cb8a3e42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34355 } } peers { permanent_uuid: "ae350714d3e54c5697b0b8e1c83ac090" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38285 } }
I20260927 12:46:11.449030 1993 heartbeater.cc:503] Master 127.0.98.62:46513 was elected leader, sending a full tablet report...
I20260927 12:46:11.449544 2017 ts_tablet_manager.cc:1428] T ca474db52dc84da497e42e2d5a5cb628 P 58e5f2ac64854c16bef4af9c707f5074: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260927 12:46:11.449808 2016 tablet_bootstrap.cc:492] T ca474db52dc84da497e42e2d5a5cb628 P 4fff3a5975fc41b488a24500cb8a3e42: No bootstrap required, opened a new log
I20260927 12:46:11.449819 2018 ts_tablet_manager.cc:1428] T ca474db52dc84da497e42e2d5a5cb628 P ae350714d3e54c5697b0b8e1c83ac090: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 12:46:11.449997 2016 ts_tablet_manager.cc:1397] T ca474db52dc84da497e42e2d5a5cb628 P 4fff3a5975fc41b488a24500cb8a3e42: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260927 12:46:11.450186 1853 heartbeater.cc:503] Master 127.0.98.62:46513 was elected leader, sending a full tablet report...
I20260927 12:46:11.453514 2016 raft_consensus.cc:348] T ca474db52dc84da497e42e2d5a5cb628 P 4fff3a5975fc41b488a24500cb8a3e42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e5f2ac64854c16bef4af9c707f5074" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44099 } } peers { permanent_uuid: "4fff3a5975fc41b488a24500cb8a3e42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34355 } } peers { permanent_uuid: "ae350714d3e54c5697b0b8e1c83ac090" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38285 } }
I20260927 12:46:11.453738 2016 raft_consensus.cc:374] T ca474db52dc84da497e42e2d5a5cb628 P 4fff3a5975fc41b488a24500cb8a3e42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:46:11.453799 2016 raft_consensus.cc:729] T ca474db52dc84da497e42e2d5a5cb628 P 4fff3a5975fc41b488a24500cb8a3e42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fff3a5975fc41b488a24500cb8a3e42, State: Initialized, Role: FOLLOWER
I20260927 12:46:11.454239 2016 consensus_queue.cc:260] T ca474db52dc84da497e42e2d5a5cb628 P 4fff3a5975fc41b488a24500cb8a3e42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e5f2ac64854c16bef4af9c707f5074" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44099 } } peers { permanent_uuid: "4fff3a5975fc41b488a24500cb8a3e42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34355 } } peers { permanent_uuid: "ae350714d3e54c5697b0b8e1c83ac090" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38285 } }
I20260927 12:46:11.454843 1713 heartbeater.cc:503] Master 127.0.98.62:46513 was elected leader, sending a full tablet report...
I20260927 12:46:11.455026 2016 ts_tablet_manager.cc:1428] T ca474db52dc84da497e42e2d5a5cb628 P 4fff3a5975fc41b488a24500cb8a3e42: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
W20260927 12:46:11.622674 1994 tablet.cc:2369] T ca474db52dc84da497e42e2d5a5cb628 P 58e5f2ac64854c16bef4af9c707f5074: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 12:46:11.647113 1854 tablet.cc:2369] T ca474db52dc84da497e42e2d5a5cb628 P ae350714d3e54c5697b0b8e1c83ac090: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:46:11.677505 2023 raft_consensus.cc:482] T ca474db52dc84da497e42e2d5a5cb628 P ae350714d3e54c5697b0b8e1c83ac090 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:46:11.677713 2023 raft_consensus.cc:504] T ca474db52dc84da497e42e2d5a5cb628 P ae350714d3e54c5697b0b8e1c83ac090 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e5f2ac64854c16bef4af9c707f5074" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44099 } } peers { permanent_uuid: "4fff3a5975fc41b488a24500cb8a3e42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34355 } } peers { permanent_uuid: "ae350714d3e54c5697b0b8e1c83ac090" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38285 } }
I20260927 12:46:11.678871 2023 leader_election.cc:288] T ca474db52dc84da497e42e2d5a5cb628 P ae350714d3e54c5697b0b8e1c83ac090 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58e5f2ac64854c16bef4af9c707f5074 (127.0.98.3:44099), 4fff3a5975fc41b488a24500cb8a3e42 (127.0.98.1:34355)
I20260927 12:46:11.682593 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:11.679270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38827 (local address 127.0.98.3:44099)
0927 12:46:11.679471 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:11.679475 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:11.679495 (+ 20us) server_negotiation.cc:407] Connection header received
0927 12:46:11.679665 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:11.679668 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:11.679708 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:11.679774 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:46:11.680025 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:11.680539 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:11.681614 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:11.682025 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:11.682139 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:11.682205 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:11.682401 (+ 196us) server_negotiation.cc:299] Negotiation successful
0927 12:46:11.682471 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
I20260927 12:46:11.682761 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:11.679245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45251 (local address 127.0.98.1:34355)
0927 12:46:11.679374 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:11.679379 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:11.679544 (+ 165us) server_negotiation.cc:407] Connection header received
0927 12:46:11.679642 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:11.679646 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:11.679694 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:11.679748 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:46:11.680003 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:11.680529 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:11.681664 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:11.682151 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:11.682250 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:11.682309 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:11.682526 (+ 217us) server_negotiation.cc:299] Negotiation successful
0927 12:46:11.682651 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 12:46:11.682621 2026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:11.679150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44099 (local address 127.0.98.2:38827)
0927 12:46:11.679435 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:11.679450 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:11.679544 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:11.679790 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:11.679793 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:11.679821 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:46:11.679929 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:11.679934 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:11.680703 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:11.680705 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:11.681512 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:11.681521 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:11.682162 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:11.682165 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:11.682257 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:11.682278 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 12:46:11.682348 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 12:46:11.682418 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"mutex_wait_us":2,"thread_start_us":82,"threads_started":1}
I20260927 12:46:11.683068 2025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:11.679209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34355 (local address 127.0.98.2:45251)
0927 12:46:11.679447 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:11.679460 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:11.679561 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:11.679771 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:11.679774 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:11.679788 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:46:11.679907 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:11.679912 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:11.680619 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:11.680622 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:11.681550 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:11.681559 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:11.682269 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:11.682272 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:11.682366 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:11.682389 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 12:46:11.682509 (+ 120us) client_negotiation.cc:240] Negotiation successful
0927 12:46:11.682574 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":94,"thread_start_us":71,"threads_started":1}
I20260927 12:46:11.683553 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca474db52dc84da497e42e2d5a5cb628" candidate_uuid: "ae350714d3e54c5697b0b8e1c83ac090" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58e5f2ac64854c16bef4af9c707f5074" is_pre_election: true
I20260927 12:46:11.683692 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca474db52dc84da497e42e2d5a5cb628" candidate_uuid: "ae350714d3e54c5697b0b8e1c83ac090" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fff3a5975fc41b488a24500cb8a3e42" is_pre_election: true
I20260927 12:46:11.683828 1947 raft_consensus.cc:2454] T ca474db52dc84da497e42e2d5a5cb628 P 58e5f2ac64854c16bef4af9c707f5074 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae350714d3e54c5697b0b8e1c83ac090 in term 0.
I20260927 12:46:11.683990 1667 raft_consensus.cc:2454] T ca474db52dc84da497e42e2d5a5cb628 P 4fff3a5975fc41b488a24500cb8a3e42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae350714d3e54c5697b0b8e1c83ac090 in term 0.
I20260927 12:46:11.684297 1738 leader_election.cc:302] T ca474db52dc84da497e42e2d5a5cb628 P ae350714d3e54c5697b0b8e1c83ac090 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58e5f2ac64854c16bef4af9c707f5074, ae350714d3e54c5697b0b8e1c83ac090; no voters:
I20260927 12:46:11.684576 2023 raft_consensus.cc:2781] T ca474db52dc84da497e42e2d5a5cb628 P ae350714d3e54c5697b0b8e1c83ac090 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:46:11.684651 2023 raft_consensus.cc:482] T ca474db52dc84da497e42e2d5a5cb628 P ae350714d3e54c5697b0b8e1c83ac090 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:46:11.684695 2023 raft_consensus.cc:3037] T ca474db52dc84da497e42e2d5a5cb628 P ae350714d3e54c5697b0b8e1c83ac090 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:46:11.685850 2023 raft_consensus.cc:504] T ca474db52dc84da497e42e2d5a5cb628 P ae350714d3e54c5697b0b8e1c83ac090 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e5f2ac64854c16bef4af9c707f5074" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44099 } } peers { permanent_uuid: "4fff3a5975fc41b488a24500cb8a3e42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34355 } } peers { permanent_uuid: "ae350714d3e54c5697b0b8e1c83ac090" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38285 } }
I20260927 12:46:11.686267 2023 leader_election.cc:288] T ca474db52dc84da497e42e2d5a5cb628 P ae350714d3e54c5697b0b8e1c83ac090 [CANDIDATE]: Term 1 election: Requested vote from peers 58e5f2ac64854c16bef4af9c707f5074 (127.0.98.3:44099), 4fff3a5975fc41b488a24500cb8a3e42 (127.0.98.1:34355)
I20260927 12:46:11.686564 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca474db52dc84da497e42e2d5a5cb628" candidate_uuid: "ae350714d3e54c5697b0b8e1c83ac090" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58e5f2ac64854c16bef4af9c707f5074"
I20260927 12:46:11.686641 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca474db52dc84da497e42e2d5a5cb628" candidate_uuid: "ae350714d3e54c5697b0b8e1c83ac090" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fff3a5975fc41b488a24500cb8a3e42"
I20260927 12:46:11.686678 1947 raft_consensus.cc:3037] T ca474db52dc84da497e42e2d5a5cb628 P 58e5f2ac64854c16bef4af9c707f5074 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:46:11.686764 1667 raft_consensus.cc:3037] T ca474db52dc84da497e42e2d5a5cb628 P 4fff3a5975fc41b488a24500cb8a3e42 [term 0 FOLLOWER]: Advancing to term 1
W20260927 12:46:11.687197 1714 tablet.cc:2369] T ca474db52dc84da497e42e2d5a5cb628 P 4fff3a5975fc41b488a24500cb8a3e42: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:46:11.687965 1667 raft_consensus.cc:2454] T ca474db52dc84da497e42e2d5a5cb628 P 4fff3a5975fc41b488a24500cb8a3e42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae350714d3e54c5697b0b8e1c83ac090 in term 1.
I20260927 12:46:11.687971 1947 raft_consensus.cc:2454] T ca474db52dc84da497e42e2d5a5cb628 P 58e5f2ac64854c16bef4af9c707f5074 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae350714d3e54c5697b0b8e1c83ac090 in term 1.
I20260927 12:46:11.688249 1740 leader_election.cc:302] T ca474db52dc84da497e42e2d5a5cb628 P ae350714d3e54c5697b0b8e1c83ac090 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fff3a5975fc41b488a24500cb8a3e42, ae350714d3e54c5697b0b8e1c83ac090; no voters:
I20260927 12:46:11.688414 2023 raft_consensus.cc:2781] T ca474db52dc84da497e42e2d5a5cb628 P ae350714d3e54c5697b0b8e1c83ac090 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:46:11.688601 2023 raft_consensus.cc:686] T ca474db52dc84da497e42e2d5a5cb628 P ae350714d3e54c5697b0b8e1c83ac090 [term 1 LEADER]: Becoming Leader. State: Replica: ae350714d3e54c5697b0b8e1c83ac090, State: Running, Role: LEADER
I20260927 12:46:11.688905 2023 consensus_queue.cc:237] T ca474db52dc84da497e42e2d5a5cb628 P ae350714d3e54c5697b0b8e1c83ac090 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e5f2ac64854c16bef4af9c707f5074" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44099 } } peers { permanent_uuid: "4fff3a5975fc41b488a24500cb8a3e42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34355 } } peers { permanent_uuid: "ae350714d3e54c5697b0b8e1c83ac090" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38285 } }
I20260927 12:46:11.691857 1509 catalog_manager.cc:5495] T ca474db52dc84da497e42e2d5a5cb628 P ae350714d3e54c5697b0b8e1c83ac090 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae350714d3e54c5697b0b8e1c83ac090 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "ae350714d3e54c5697b0b8e1c83ac090" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e5f2ac64854c16bef4af9c707f5074" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fff3a5975fc41b488a24500cb8a3e42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae350714d3e54c5697b0b8e1c83ac090" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38285 } health_report { overall_health: HEALTHY } } }
I20260927 12:46:11.727646 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:11.724231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47494 (local address 127.0.98.2:38285)
0927 12:46:11.724352 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:11.724356 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:11.724367 (+ 11us) server_negotiation.cc:407] Connection header received
0927 12:46:11.724432 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:11.724434 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:11.724486 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:11.724574 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:46:11.724843 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:11.725362 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:11.726244 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:11.726452 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:11.726536 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:11.726858 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:46:11.727273 (+ 415us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:46:11.727369 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:11.727456 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 12:46:11.727510 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 12:46:11.733287 1667 raft_consensus.cc:1261] T ca474db52dc84da497e42e2d5a5cb628 P 4fff3a5975fc41b488a24500cb8a3e42 [term 1 FOLLOWER]: Refusing update from remote peer ae350714d3e54c5697b0b8e1c83ac090: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:46:11.733289 1947 raft_consensus.cc:1261] T ca474db52dc84da497e42e2d5a5cb628 P 58e5f2ac64854c16bef4af9c707f5074 [term 1 FOLLOWER]: Refusing update from remote peer ae350714d3e54c5697b0b8e1c83ac090: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:46:11.733994 2027 consensus_queue.cc:1035] T ca474db52dc84da497e42e2d5a5cb628 P ae350714d3e54c5697b0b8e1c83ac090 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58e5f2ac64854c16bef4af9c707f5074" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:46:11.734195 2023 consensus_queue.cc:1035] T ca474db52dc84da497e42e2d5a5cb628 P ae350714d3e54c5697b0b8e1c83ac090 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fff3a5975fc41b488a24500cb8a3e42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:46:11.741950 2031 mvcc.cc:205] Tried to move back new op lower bound from 7333941951412740096 to 7333941951240282112. Current Snapshot: MvccSnapshot[applied={T|T < 7333941951412740096}]
I20260927 12:46:11.742480 2032 mvcc.cc:205] Tried to move back new op lower bound from 7333941951412740096 to 7333941951240282112. Current Snapshot: MvccSnapshot[applied={T|T < 7333941951412740096}]
I20260927 12:46:11.753455 1509 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:46:11.753625 1509 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:46:11.756158 1509 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f8fd783ac9124faa86455d32e73f8d7a: Sending DeleteTablet for 3 replicas of tablet ca474db52dc84da497e42e2d5a5cb628
I20260927 12:46:11.756989 1927 tablet_service.cc:1514] Processing DeleteTablet for tablet ca474db52dc84da497e42e2d5a5cb628 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:46:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37512
I20260927 12:46:11.757133 1647 tablet_service.cc:1514] Processing DeleteTablet for tablet ca474db52dc84da497e42e2d5a5cb628 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:46:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35128
I20260927 12:46:11.757596 2041 tablet_replica.cc:330] T ca474db52dc84da497e42e2d5a5cb628 P 4fff3a5975fc41b488a24500cb8a3e42: stopping tablet replica
I20260927 12:46:11.757972 2041 raft_consensus.cc:2229] T ca474db52dc84da497e42e2d5a5cb628 P 4fff3a5975fc41b488a24500cb8a3e42 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:46:11.758033 1787 tablet_service.cc:1514] Processing DeleteTablet for tablet ca474db52dc84da497e42e2d5a5cb628 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:46:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38360
I20260927 12:46:11.758337 2041 raft_consensus.cc:2258] T ca474db52dc84da497e42e2d5a5cb628 P 4fff3a5975fc41b488a24500cb8a3e42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:46:11.758335 2040 tablet_replica.cc:330] T ca474db52dc84da497e42e2d5a5cb628 P 58e5f2ac64854c16bef4af9c707f5074: stopping tablet replica
I20260927 12:46:11.758549 1509 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58404:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:46:11.758980 1509 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:46:11.759027 2040 raft_consensus.cc:2229] T ca474db52dc84da497e42e2d5a5cb628 P 58e5f2ac64854c16bef4af9c707f5074 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:46:11.759454 2040 raft_consensus.cc:2258] T ca474db52dc84da497e42e2d5a5cb628 P 58e5f2ac64854c16bef4af9c707f5074 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:46:11.759874 2041 ts_tablet_manager.cc:1905] T ca474db52dc84da497e42e2d5a5cb628 P 4fff3a5975fc41b488a24500cb8a3e42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:46:11.760679 2042 tablet_replica.cc:330] T ca474db52dc84da497e42e2d5a5cb628 P ae350714d3e54c5697b0b8e1c83ac090: stopping tablet replica
I20260927 12:46:11.760962 2042 raft_consensus.cc:2229] T ca474db52dc84da497e42e2d5a5cb628 P ae350714d3e54c5697b0b8e1c83ac090 [term 1 LEADER]: Raft consensus shutting down.
I20260927 12:46:11.761397 2042 raft_consensus.cc:2258] T ca474db52dc84da497e42e2d5a5cb628 P ae350714d3e54c5697b0b8e1c83ac090 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:46:11.762651 2041 ts_tablet_manager.cc:1918] T ca474db52dc84da497e42e2d5a5cb628 P 4fff3a5975fc41b488a24500cb8a3e42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:46:11.762753 2041 log.cc:1196] T ca474db52dc84da497e42e2d5a5cb628 P 4fff3a5975fc41b488a24500cb8a3e42: Deleting WAL directory at /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/minicluster-data/ts-0/wal/wals/ca474db52dc84da497e42e2d5a5cb628
I20260927 12:46:11.763065 2041 ts_tablet_manager.cc:1939] T ca474db52dc84da497e42e2d5a5cb628 P 4fff3a5975fc41b488a24500cb8a3e42: Deleting consensus metadata
I20260927 12:46:11.763921 2042 ts_tablet_manager.cc:1905] T ca474db52dc84da497e42e2d5a5cb628 P ae350714d3e54c5697b0b8e1c83ac090: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:46:11.764380 2040 ts_tablet_manager.cc:1905] T ca474db52dc84da497e42e2d5a5cb628 P 58e5f2ac64854c16bef4af9c707f5074: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:46:11.764549 1496 catalog_manager.cc:4841] TS 4fff3a5975fc41b488a24500cb8a3e42 (127.0.98.1:34355): tablet ca474db52dc84da497e42e2d5a5cb628 (table test-table [id=17ae78770d2f431ca16ac10bb0dc9e97]) successfully deleted
I20260927 12:46:11.764892 1927 tablet_service.cc:1467] Processing CreateTablet for tablet 99cc3b6d1004452682548ab91f4abfab (DEFAULT_TABLE table=test-table [id=1a8aff8e653342fc9dde2fd82dde2c0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:46:11.765043 1647 tablet_service.cc:1467] Processing CreateTablet for tablet 99cc3b6d1004452682548ab91f4abfab (DEFAULT_TABLE table=test-table [id=1a8aff8e653342fc9dde2fd82dde2c0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:46:11.765139 1927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99cc3b6d1004452682548ab91f4abfab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:46:11.765283 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99cc3b6d1004452682548ab91f4abfab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:46:11.767417 2016 tablet_bootstrap.cc:492] T 99cc3b6d1004452682548ab91f4abfab P 4fff3a5975fc41b488a24500cb8a3e42: Bootstrap starting.
I20260927 12:46:11.767449 2017 tablet_bootstrap.cc:492] T 99cc3b6d1004452682548ab91f4abfab P 58e5f2ac64854c16bef4af9c707f5074: Bootstrap starting.
I20260927 12:46:11.768028 1787 tablet_service.cc:1467] Processing CreateTablet for tablet 99cc3b6d1004452682548ab91f4abfab (DEFAULT_TABLE table=test-table [id=1a8aff8e653342fc9dde2fd82dde2c0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:46:11.768251 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99cc3b6d1004452682548ab91f4abfab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:46:11.768307 2017 tablet_bootstrap.cc:654] T 99cc3b6d1004452682548ab91f4abfab P 58e5f2ac64854c16bef4af9c707f5074: Neither blocks nor log segments found. Creating new log.
I20260927 12:46:11.768316 2016 tablet_bootstrap.cc:654] T 99cc3b6d1004452682548ab91f4abfab P 4fff3a5975fc41b488a24500cb8a3e42: Neither blocks nor log segments found. Creating new log.
I20260927 12:46:11.769677 2017 tablet_bootstrap.cc:492] T 99cc3b6d1004452682548ab91f4abfab P 58e5f2ac64854c16bef4af9c707f5074: No bootstrap required, opened a new log
I20260927 12:46:11.769763 2017 ts_tablet_manager.cc:1397] T 99cc3b6d1004452682548ab91f4abfab P 58e5f2ac64854c16bef4af9c707f5074: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260927 12:46:11.770298 2018 tablet_bootstrap.cc:492] T 99cc3b6d1004452682548ab91f4abfab P ae350714d3e54c5697b0b8e1c83ac090: Bootstrap starting.
I20260927 12:46:11.770242 2017 raft_consensus.cc:348] T 99cc3b6d1004452682548ab91f4abfab P 58e5f2ac64854c16bef4af9c707f5074 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fff3a5975fc41b488a24500cb8a3e42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34355 } } peers { permanent_uuid: "ae350714d3e54c5697b0b8e1c83ac090" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38285 } } peers { permanent_uuid: "58e5f2ac64854c16bef4af9c707f5074" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44099 } }
I20260927 12:46:11.770350 2017 raft_consensus.cc:374] T 99cc3b6d1004452682548ab91f4abfab P 58e5f2ac64854c16bef4af9c707f5074 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:46:11.770387 2017 raft_consensus.cc:729] T 99cc3b6d1004452682548ab91f4abfab P 58e5f2ac64854c16bef4af9c707f5074 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58e5f2ac64854c16bef4af9c707f5074, State: Initialized, Role: FOLLOWER
I20260927 12:46:11.770502 2017 consensus_queue.cc:260] T 99cc3b6d1004452682548ab91f4abfab P 58e5f2ac64854c16bef4af9c707f5074 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fff3a5975fc41b488a24500cb8a3e42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34355 } } peers { permanent_uuid: "ae350714d3e54c5697b0b8e1c83ac090" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38285 } } peers { permanent_uuid: "58e5f2ac64854c16bef4af9c707f5074" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44099 } }
I20260927 12:46:11.770746 2017 ts_tablet_manager.cc:1428] T 99cc3b6d1004452682548ab91f4abfab P 58e5f2ac64854c16bef4af9c707f5074: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 12:46:11.771121 2040 ts_tablet_manager.cc:1918] T ca474db52dc84da497e42e2d5a5cb628 P 58e5f2ac64854c16bef4af9c707f5074: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:46:11.771265 2018 tablet_bootstrap.cc:654] T 99cc3b6d1004452682548ab91f4abfab P ae350714d3e54c5697b0b8e1c83ac090: Neither blocks nor log segments found. Creating new log.
I20260927 12:46:11.771400 2040 log.cc:1196] T ca474db52dc84da497e42e2d5a5cb628 P 58e5f2ac64854c16bef4af9c707f5074: Deleting WAL directory at /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/minicluster-data/ts-2/wal/wals/ca474db52dc84da497e42e2d5a5cb628
I20260927 12:46:11.771816 2042 ts_tablet_manager.cc:1918] T ca474db52dc84da497e42e2d5a5cb628 P ae350714d3e54c5697b0b8e1c83ac090: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:46:11.772198 2040 ts_tablet_manager.cc:1939] T ca474db52dc84da497e42e2d5a5cb628 P 58e5f2ac64854c16bef4af9c707f5074: Deleting consensus metadata
I20260927 12:46:11.772502 2018 tablet_bootstrap.cc:492] T 99cc3b6d1004452682548ab91f4abfab P ae350714d3e54c5697b0b8e1c83ac090: No bootstrap required, opened a new log
I20260927 12:46:11.773355 2018 ts_tablet_manager.cc:1397] T 99cc3b6d1004452682548ab91f4abfab P ae350714d3e54c5697b0b8e1c83ac090: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 12:46:11.773383 1494 catalog_manager.cc:4841] TS 58e5f2ac64854c16bef4af9c707f5074 (127.0.98.3:44099): tablet ca474db52dc84da497e42e2d5a5cb628 (table test-table [id=17ae78770d2f431ca16ac10bb0dc9e97]) successfully deleted
I20260927 12:46:11.773768 2018 raft_consensus.cc:348] T 99cc3b6d1004452682548ab91f4abfab P ae350714d3e54c5697b0b8e1c83ac090 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fff3a5975fc41b488a24500cb8a3e42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34355 } } peers { permanent_uuid: "ae350714d3e54c5697b0b8e1c83ac090" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38285 } } peers { permanent_uuid: "58e5f2ac64854c16bef4af9c707f5074" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44099 } }
I20260927 12:46:11.773866 2018 raft_consensus.cc:374] T 99cc3b6d1004452682548ab91f4abfab P ae350714d3e54c5697b0b8e1c83ac090 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:46:11.773897 2018 raft_consensus.cc:729] T 99cc3b6d1004452682548ab91f4abfab P ae350714d3e54c5697b0b8e1c83ac090 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae350714d3e54c5697b0b8e1c83ac090, State: Initialized, Role: FOLLOWER
I20260927 12:46:11.773998 2018 consensus_queue.cc:260] T 99cc3b6d1004452682548ab91f4abfab P ae350714d3e54c5697b0b8e1c83ac090 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fff3a5975fc41b488a24500cb8a3e42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34355 } } peers { permanent_uuid: "ae350714d3e54c5697b0b8e1c83ac090" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38285 } } peers { permanent_uuid: "58e5f2ac64854c16bef4af9c707f5074" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44099 } }
I20260927 12:46:11.774168 2042 log.cc:1196] T ca474db52dc84da497e42e2d5a5cb628 P ae350714d3e54c5697b0b8e1c83ac090: Deleting WAL directory at /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/minicluster-data/ts-1/wal/wals/ca474db52dc84da497e42e2d5a5cb628
I20260927 12:46:11.774251 2018 ts_tablet_manager.cc:1428] T 99cc3b6d1004452682548ab91f4abfab P ae350714d3e54c5697b0b8e1c83ac090: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 12:46:11.774573 2042 ts_tablet_manager.cc:1939] T ca474db52dc84da497e42e2d5a5cb628 P ae350714d3e54c5697b0b8e1c83ac090: Deleting consensus metadata
I20260927 12:46:11.775086 2016 tablet_bootstrap.cc:492] T 99cc3b6d1004452682548ab91f4abfab P 4fff3a5975fc41b488a24500cb8a3e42: No bootstrap required, opened a new log
I20260927 12:46:11.775177 2016 ts_tablet_manager.cc:1397] T 99cc3b6d1004452682548ab91f4abfab P 4fff3a5975fc41b488a24500cb8a3e42: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.001s
I20260927 12:46:11.775704 1495 catalog_manager.cc:4841] TS ae350714d3e54c5697b0b8e1c83ac090 (127.0.98.2:38285): tablet ca474db52dc84da497e42e2d5a5cb628 (table test-table [id=17ae78770d2f431ca16ac10bb0dc9e97]) successfully deleted
I20260927 12:46:11.775876 2016 raft_consensus.cc:348] T 99cc3b6d1004452682548ab91f4abfab P 4fff3a5975fc41b488a24500cb8a3e42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fff3a5975fc41b488a24500cb8a3e42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34355 } } peers { permanent_uuid: "ae350714d3e54c5697b0b8e1c83ac090" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38285 } } peers { permanent_uuid: "58e5f2ac64854c16bef4af9c707f5074" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44099 } }
I20260927 12:46:11.775985 2016 raft_consensus.cc:374] T 99cc3b6d1004452682548ab91f4abfab P 4fff3a5975fc41b488a24500cb8a3e42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:46:11.776013 2016 raft_consensus.cc:729] T 99cc3b6d1004452682548ab91f4abfab P 4fff3a5975fc41b488a24500cb8a3e42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fff3a5975fc41b488a24500cb8a3e42, State: Initialized, Role: FOLLOWER
I20260927 12:46:11.776121 2016 consensus_queue.cc:260] T 99cc3b6d1004452682548ab91f4abfab P 4fff3a5975fc41b488a24500cb8a3e42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fff3a5975fc41b488a24500cb8a3e42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34355 } } peers { permanent_uuid: "ae350714d3e54c5697b0b8e1c83ac090" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38285 } } peers { permanent_uuid: "58e5f2ac64854c16bef4af9c707f5074" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44099 } }
I20260927 12:46:11.776381 2016 ts_tablet_manager.cc:1428] T 99cc3b6d1004452682548ab91f4abfab P 4fff3a5975fc41b488a24500cb8a3e42: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260927 12:46:11.873474 2027 raft_consensus.cc:482] T 99cc3b6d1004452682548ab91f4abfab P ae350714d3e54c5697b0b8e1c83ac090 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:46:11.873639 2027 raft_consensus.cc:504] T 99cc3b6d1004452682548ab91f4abfab P ae350714d3e54c5697b0b8e1c83ac090 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fff3a5975fc41b488a24500cb8a3e42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34355 } } peers { permanent_uuid: "ae350714d3e54c5697b0b8e1c83ac090" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38285 } } peers { permanent_uuid: "58e5f2ac64854c16bef4af9c707f5074" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44099 } }
I20260927 12:46:11.874145 2027 leader_election.cc:288] T 99cc3b6d1004452682548ab91f4abfab P ae350714d3e54c5697b0b8e1c83ac090 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fff3a5975fc41b488a24500cb8a3e42 (127.0.98.1:34355), 58e5f2ac64854c16bef4af9c707f5074 (127.0.98.3:44099)
I20260927 12:46:11.874578 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99cc3b6d1004452682548ab91f4abfab" candidate_uuid: "ae350714d3e54c5697b0b8e1c83ac090" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fff3a5975fc41b488a24500cb8a3e42" is_pre_election: true
I20260927 12:46:11.874578 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99cc3b6d1004452682548ab91f4abfab" candidate_uuid: "ae350714d3e54c5697b0b8e1c83ac090" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58e5f2ac64854c16bef4af9c707f5074" is_pre_election: true
I20260927 12:46:11.874730 1947 raft_consensus.cc:2454] T 99cc3b6d1004452682548ab91f4abfab P 58e5f2ac64854c16bef4af9c707f5074 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae350714d3e54c5697b0b8e1c83ac090 in term 0.
I20260927 12:46:11.874730 1667 raft_consensus.cc:2454] T 99cc3b6d1004452682548ab91f4abfab P 4fff3a5975fc41b488a24500cb8a3e42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae350714d3e54c5697b0b8e1c83ac090 in term 0.
I20260927 12:46:11.875025 1740 leader_election.cc:302] T 99cc3b6d1004452682548ab91f4abfab P ae350714d3e54c5697b0b8e1c83ac090 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fff3a5975fc41b488a24500cb8a3e42, ae350714d3e54c5697b0b8e1c83ac090; no voters:
I20260927 12:46:11.875193 2027 raft_consensus.cc:2781] T 99cc3b6d1004452682548ab91f4abfab P ae350714d3e54c5697b0b8e1c83ac090 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:46:11.875254 2027 raft_consensus.cc:482] T 99cc3b6d1004452682548ab91f4abfab P ae350714d3e54c5697b0b8e1c83ac090 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:46:11.875306 2027 raft_consensus.cc:3037] T 99cc3b6d1004452682548ab91f4abfab P ae350714d3e54c5697b0b8e1c83ac090 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:46:11.876407 2027 raft_consensus.cc:504] T 99cc3b6d1004452682548ab91f4abfab P ae350714d3e54c5697b0b8e1c83ac090 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fff3a5975fc41b488a24500cb8a3e42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34355 } } peers { permanent_uuid: "ae350714d3e54c5697b0b8e1c83ac090" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38285 } } peers { permanent_uuid: "58e5f2ac64854c16bef4af9c707f5074" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44099 } }
I20260927 12:46:11.876904 2027 leader_election.cc:288] T 99cc3b6d1004452682548ab91f4abfab P ae350714d3e54c5697b0b8e1c83ac090 [CANDIDATE]: Term 1 election: Requested vote from peers 4fff3a5975fc41b488a24500cb8a3e42 (127.0.98.1:34355), 58e5f2ac64854c16bef4af9c707f5074 (127.0.98.3:44099)
I20260927 12:46:11.877213 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99cc3b6d1004452682548ab91f4abfab" candidate_uuid: "ae350714d3e54c5697b0b8e1c83ac090" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fff3a5975fc41b488a24500cb8a3e42"
I20260927 12:46:11.877271 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99cc3b6d1004452682548ab91f4abfab" candidate_uuid: "ae350714d3e54c5697b0b8e1c83ac090" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58e5f2ac64854c16bef4af9c707f5074"
I20260927 12:46:11.877372 1667 raft_consensus.cc:3037] T 99cc3b6d1004452682548ab91f4abfab P 4fff3a5975fc41b488a24500cb8a3e42 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:46:11.877372 1947 raft_consensus.cc:3037] T 99cc3b6d1004452682548ab91f4abfab P 58e5f2ac64854c16bef4af9c707f5074 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:46:11.878587 1947 raft_consensus.cc:2454] T 99cc3b6d1004452682548ab91f4abfab P 58e5f2ac64854c16bef4af9c707f5074 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae350714d3e54c5697b0b8e1c83ac090 in term 1.
I20260927 12:46:11.878589 1667 raft_consensus.cc:2454] T 99cc3b6d1004452682548ab91f4abfab P 4fff3a5975fc41b488a24500cb8a3e42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae350714d3e54c5697b0b8e1c83ac090 in term 1.
I20260927 12:46:11.878880 1740 leader_election.cc:302] T 99cc3b6d1004452682548ab91f4abfab P ae350714d3e54c5697b0b8e1c83ac090 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fff3a5975fc41b488a24500cb8a3e42, ae350714d3e54c5697b0b8e1c83ac090; no voters:
I20260927 12:46:11.879035 2027 raft_consensus.cc:2781] T 99cc3b6d1004452682548ab91f4abfab P ae350714d3e54c5697b0b8e1c83ac090 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:46:11.879118 2027 raft_consensus.cc:686] T 99cc3b6d1004452682548ab91f4abfab P ae350714d3e54c5697b0b8e1c83ac090 [term 1 LEADER]: Becoming Leader. State: Replica: ae350714d3e54c5697b0b8e1c83ac090, State: Running, Role: LEADER
I20260927 12:46:11.879272 2027 consensus_queue.cc:237] T 99cc3b6d1004452682548ab91f4abfab P ae350714d3e54c5697b0b8e1c83ac090 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fff3a5975fc41b488a24500cb8a3e42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34355 } } peers { permanent_uuid: "ae350714d3e54c5697b0b8e1c83ac090" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38285 } } peers { permanent_uuid: "58e5f2ac64854c16bef4af9c707f5074" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44099 } }
I20260927 12:46:11.880842 1508 catalog_manager.cc:5495] T 99cc3b6d1004452682548ab91f4abfab P ae350714d3e54c5697b0b8e1c83ac090 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae350714d3e54c5697b0b8e1c83ac090 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "ae350714d3e54c5697b0b8e1c83ac090" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fff3a5975fc41b488a24500cb8a3e42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae350714d3e54c5697b0b8e1c83ac090" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58e5f2ac64854c16bef4af9c707f5074" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44099 } health_report { overall_health: UNKNOWN } } }
Sep 27 12:46:11 dist-test-slave-lkn4 krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513170, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:46:11.919889 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:11.911684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46513 (local address 127.0.0.1:58394)
0927 12:46:11.911971 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:11.911984 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:11.912081 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:11.912305 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:11.912307 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:11.912761 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:11.912862 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:11.912868 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:11.913596 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:11.913599 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:11.914215 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:11.914224 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:11.914742 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:11.914745 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:11.914809 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:11.915380 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:11.915391 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:11.916892 (+ 1501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:11.918453 (+ 1561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:11.918460 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:11.918465 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:11.918693 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:11.918975 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:11.918977 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:11.918978 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:11.919052 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:11.919388 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:11.919393 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:11.919495 (+ 102us) client_negotiation.cc:769] Sending connection context
0927 12:46:11.919589 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 12:46:11.919750 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":106,"threads_started":1}
I20260927 12:46:11.919993 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:11.911725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58394 (local address 127.0.98.62:46513)
0927 12:46:11.912061 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:11.912067 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:11.912081 (+ 14us) server_negotiation.cc:407] Connection header received
0927 12:46:11.912162 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:11.912165 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:11.912202 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:11.912274 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:11.912967 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:11.913502 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:11.914317 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:11.914548 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:11.914624 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:11.917017 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:11.917020 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:11.917022 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:11.917041 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:11.918367 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:11.918815 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:11.918818 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:11.918820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:11.918898 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:11.919126 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:11.919130 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:11.919132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:11.919314 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:11.919387 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:11.919733 (+ 346us) server_negotiation.cc:299] Negotiation successful
0927 12:46:11.919832 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":87,"threads_started":1}
I20260927 12:46:11.923861 1508 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58394:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 12:46:11.924538 1508 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:46:11.929215 1787 tablet_service.cc:1467] Processing CreateTablet for tablet 893a0b4abfbf4ba8bcbdea8c44816ab2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5729794748164a58a9455912d5f3be3b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 12:46:11.929263 1927 tablet_service.cc:1467] Processing CreateTablet for tablet 893a0b4abfbf4ba8bcbdea8c44816ab2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5729794748164a58a9455912d5f3be3b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 12:46:11.929216 1647 tablet_service.cc:1467] Processing CreateTablet for tablet 893a0b4abfbf4ba8bcbdea8c44816ab2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5729794748164a58a9455912d5f3be3b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 12:46:11.929592 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 893a0b4abfbf4ba8bcbdea8c44816ab2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:46:11.929596 1927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 893a0b4abfbf4ba8bcbdea8c44816ab2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:46:11.929596 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 893a0b4abfbf4ba8bcbdea8c44816ab2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:46:11.931967 2016 tablet_bootstrap.cc:492] T 893a0b4abfbf4ba8bcbdea8c44816ab2 P 4fff3a5975fc41b488a24500cb8a3e42: Bootstrap starting.
I20260927 12:46:11.932185 2018 tablet_bootstrap.cc:492] T 893a0b4abfbf4ba8bcbdea8c44816ab2 P ae350714d3e54c5697b0b8e1c83ac090: Bootstrap starting.
I20260927 12:46:11.933072 2018 tablet_bootstrap.cc:654] T 893a0b4abfbf4ba8bcbdea8c44816ab2 P ae350714d3e54c5697b0b8e1c83ac090: Neither blocks nor log segments found. Creating new log.
I20260927 12:46:11.933435 2016 tablet_bootstrap.cc:654] T 893a0b4abfbf4ba8bcbdea8c44816ab2 P 4fff3a5975fc41b488a24500cb8a3e42: Neither blocks nor log segments found. Creating new log.
I20260927 12:46:11.934091 2017 tablet_bootstrap.cc:492] T 893a0b4abfbf4ba8bcbdea8c44816ab2 P 58e5f2ac64854c16bef4af9c707f5074: Bootstrap starting.
I20260927 12:46:11.934224 2018 tablet_bootstrap.cc:492] T 893a0b4abfbf4ba8bcbdea8c44816ab2 P ae350714d3e54c5697b0b8e1c83ac090: No bootstrap required, opened a new log
I20260927 12:46:11.934302 2018 ts_tablet_manager.cc:1397] T 893a0b4abfbf4ba8bcbdea8c44816ab2 P ae350714d3e54c5697b0b8e1c83ac090: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 12:46:11.934728 2018 raft_consensus.cc:348] T 893a0b4abfbf4ba8bcbdea8c44816ab2 P ae350714d3e54c5697b0b8e1c83ac090 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fff3a5975fc41b488a24500cb8a3e42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34355 } } peers { permanent_uuid: "ae350714d3e54c5697b0b8e1c83ac090" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38285 } } peers { permanent_uuid: "58e5f2ac64854c16bef4af9c707f5074" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44099 } }
I20260927 12:46:11.934844 2018 raft_consensus.cc:374] T 893a0b4abfbf4ba8bcbdea8c44816ab2 P ae350714d3e54c5697b0b8e1c83ac090 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:46:11.934873 2018 raft_consensus.cc:729] T 893a0b4abfbf4ba8bcbdea8c44816ab2 P ae350714d3e54c5697b0b8e1c83ac090 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae350714d3e54c5697b0b8e1c83ac090, State: Initialized, Role: FOLLOWER
I20260927 12:46:11.934942 2017 tablet_bootstrap.cc:654] T 893a0b4abfbf4ba8bcbdea8c44816ab2 P 58e5f2ac64854c16bef4af9c707f5074: Neither blocks nor log segments found. Creating new log.
I20260927 12:46:11.934989 2018 consensus_queue.cc:260] T 893a0b4abfbf4ba8bcbdea8c44816ab2 P ae350714d3e54c5697b0b8e1c83ac090 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fff3a5975fc41b488a24500cb8a3e42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34355 } } peers { permanent_uuid: "ae350714d3e54c5697b0b8e1c83ac090" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38285 } } peers { permanent_uuid: "58e5f2ac64854c16bef4af9c707f5074" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44099 } }
I20260927 12:46:11.935259 2018 ts_tablet_manager.cc:1428] T 893a0b4abfbf4ba8bcbdea8c44816ab2 P ae350714d3e54c5697b0b8e1c83ac090: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 12:46:11.935216 2027 tablet_replica.cc:439] T 893a0b4abfbf4ba8bcbdea8c44816ab2 P ae350714d3e54c5697b0b8e1c83ac090: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fff3a5975fc41b488a24500cb8a3e42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34355 } } peers { permanent_uuid: "ae350714d3e54c5697b0b8e1c83ac090" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38285 } } peers { permanent_uuid: "58e5f2ac64854c16bef4af9c707f5074" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44099 } } }
I20260927 12:46:11.935348 2027 tablet_replica.cc:442] T 893a0b4abfbf4ba8bcbdea8c44816ab2 P ae350714d3e54c5697b0b8e1c83ac090: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 12:46:11.935866 2016 tablet_bootstrap.cc:492] T 893a0b4abfbf4ba8bcbdea8c44816ab2 P 4fff3a5975fc41b488a24500cb8a3e42: No bootstrap required, opened a new log
I20260927 12:46:11.935940 2016 ts_tablet_manager.cc:1397] T 893a0b4abfbf4ba8bcbdea8c44816ab2 P 4fff3a5975fc41b488a24500cb8a3e42: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260927 12:46:11.936326 2016 raft_consensus.cc:348] T 893a0b4abfbf4ba8bcbdea8c44816ab2 P 4fff3a5975fc41b488a24500cb8a3e42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fff3a5975fc41b488a24500cb8a3e42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34355 } } peers { permanent_uuid: "ae350714d3e54c5697b0b8e1c83ac090" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38285 } } peers { permanent_uuid: "58e5f2ac64854c16bef4af9c707f5074" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44099 } }
I20260927 12:46:11.936430 2016 raft_consensus.cc:374] T 893a0b4abfbf4ba8bcbdea8c44816ab2 P 4fff3a5975fc41b488a24500cb8a3e42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:46:11.936462 2016 raft_consensus.cc:729] T 893a0b4abfbf4ba8bcbdea8c44816ab2 P 4fff3a5975fc41b488a24500cb8a3e42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fff3a5975fc41b488a24500cb8a3e42, State: Initialized, Role: FOLLOWER
I20260927 12:46:11.936911 2017 tablet_bootstrap.cc:492] T 893a0b4abfbf4ba8bcbdea8c44816ab2 P 58e5f2ac64854c16bef4af9c707f5074: No bootstrap required, opened a new log
I20260927 12:46:11.936980 2017 ts_tablet_manager.cc:1397] T 893a0b4abfbf4ba8bcbdea8c44816ab2 P 58e5f2ac64854c16bef4af9c707f5074: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 12:46:11.937330 2017 raft_consensus.cc:348] T 893a0b4abfbf4ba8bcbdea8c44816ab2 P 58e5f2ac64854c16bef4af9c707f5074 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fff3a5975fc41b488a24500cb8a3e42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34355 } } peers { permanent_uuid: "ae350714d3e54c5697b0b8e1c83ac090" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38285 } } peers { permanent_uuid: "58e5f2ac64854c16bef4af9c707f5074" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44099 } }
I20260927 12:46:11.937417 2017 raft_consensus.cc:374] T 893a0b4abfbf4ba8bcbdea8c44816ab2 P 58e5f2ac64854c16bef4af9c707f5074 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:46:11.937438 2017 raft_consensus.cc:729] T 893a0b4abfbf4ba8bcbdea8c44816ab2 P 58e5f2ac64854c16bef4af9c707f5074 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58e5f2ac64854c16bef4af9c707f5074, State: Initialized, Role: FOLLOWER
I20260927 12:46:11.937525 2017 consensus_queue.cc:260] T 893a0b4abfbf4ba8bcbdea8c44816ab2 P 58e5f2ac64854c16bef4af9c707f5074 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fff3a5975fc41b488a24500cb8a3e42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34355 } } peers { permanent_uuid: "ae350714d3e54c5697b0b8e1c83ac090" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38285 } } peers { permanent_uuid: "58e5f2ac64854c16bef4af9c707f5074" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44099 } }
I20260927 12:46:11.937726 2017 ts_tablet_manager.cc:1428] T 893a0b4abfbf4ba8bcbdea8c44816ab2 P 58e5f2ac64854c16bef4af9c707f5074: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 12:46:11.937732 2022 tablet_replica.cc:439] T 893a0b4abfbf4ba8bcbdea8c44816ab2 P 58e5f2ac64854c16bef4af9c707f5074: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fff3a5975fc41b488a24500cb8a3e42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34355 } } peers { permanent_uuid: "ae350714d3e54c5697b0b8e1c83ac090" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38285 } } peers { permanent_uuid: "58e5f2ac64854c16bef4af9c707f5074" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44099 } } }
I20260927 12:46:11.937891 2022 tablet_replica.cc:442] T 893a0b4abfbf4ba8bcbdea8c44816ab2 P 58e5f2ac64854c16bef4af9c707f5074: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 12:46:11.937875 2016 consensus_queue.cc:260] T 893a0b4abfbf4ba8bcbdea8c44816ab2 P 4fff3a5975fc41b488a24500cb8a3e42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fff3a5975fc41b488a24500cb8a3e42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34355 } } peers { permanent_uuid: "ae350714d3e54c5697b0b8e1c83ac090" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38285 } } peers { permanent_uuid: "58e5f2ac64854c16bef4af9c707f5074" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44099 } }
I20260927 12:46:11.938145 2024 tablet_replica.cc:439] T 893a0b4abfbf4ba8bcbdea8c44816ab2 P 4fff3a5975fc41b488a24500cb8a3e42: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fff3a5975fc41b488a24500cb8a3e42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34355 } } peers { permanent_uuid: "ae350714d3e54c5697b0b8e1c83ac090" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38285 } } peers { permanent_uuid: "58e5f2ac64854c16bef4af9c707f5074" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44099 } } }
I20260927 12:46:11.938256 2024 tablet_replica.cc:442] T 893a0b4abfbf4ba8bcbdea8c44816ab2 P 4fff3a5975fc41b488a24500cb8a3e42: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 12:46:11.939030 2016 ts_tablet_manager.cc:1428] T 893a0b4abfbf4ba8bcbdea8c44816ab2 P 4fff3a5975fc41b488a24500cb8a3e42: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260927 12:46:12.293190 2027 consensus_queue.cc:1035] T 99cc3b6d1004452682548ab91f4abfab P ae350714d3e54c5697b0b8e1c83ac090 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58e5f2ac64854c16bef4af9c707f5074" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:46:12.296520 2027 consensus_queue.cc:1035] T 99cc3b6d1004452682548ab91f4abfab P ae350714d3e54c5697b0b8e1c83ac090 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fff3a5975fc41b488a24500cb8a3e42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:46:12.331970 2024 raft_consensus.cc:482] T 893a0b4abfbf4ba8bcbdea8c44816ab2 P 4fff3a5975fc41b488a24500cb8a3e42 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:46:12.332115 2024 raft_consensus.cc:504] T 893a0b4abfbf4ba8bcbdea8c44816ab2 P 4fff3a5975fc41b488a24500cb8a3e42 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fff3a5975fc41b488a24500cb8a3e42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34355 } } peers { permanent_uuid: "ae350714d3e54c5697b0b8e1c83ac090" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38285 } } peers { permanent_uuid: "58e5f2ac64854c16bef4af9c707f5074" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44099 } }
I20260927 12:46:12.333226 2024 leader_election.cc:288] T 893a0b4abfbf4ba8bcbdea8c44816ab2 P 4fff3a5975fc41b488a24500cb8a3e42 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae350714d3e54c5697b0b8e1c83ac090 (127.0.98.2:38285), 58e5f2ac64854c16bef4af9c707f5074 (127.0.98.3:44099)
I20260927 12:46:12.337141 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:12.333549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49803 (local address 127.0.98.2:38285)
0927 12:46:12.333882 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:12.333886 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:12.333922 (+ 36us) server_negotiation.cc:407] Connection header received
0927 12:46:12.334067 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:12.334069 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:12.334119 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:12.334212 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:46:12.334603 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:12.335132 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:12.336194 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:12.336599 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:12.336676 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:12.336747 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:12.336923 (+ 176us) server_negotiation.cc:299] Negotiation successful
0927 12:46:12.337001 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":66,"threads_started":1}
I20260927 12:46:12.337141 2066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:12.333587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53145 (local address 127.0.98.3:44099)
0927 12:46:12.333835 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:12.333840 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:12.333883 (+ 43us) server_negotiation.cc:407] Connection header received
0927 12:46:12.334068 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:12.334070 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:12.334119 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:12.334218 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:46:12.334574 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:12.335161 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:12.336192 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:12.336641 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:12.336747 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:12.336809 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:12.336945 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 12:46:12.337015 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":89,"threads_started":1}
I20260927 12:46:12.337144 2064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:12.333524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44099 (local address 127.0.98.1:53145)
0927 12:46:12.333807 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:12.333824 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:12.333949 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:12.334230 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:12.334232 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:12.334268 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:46:12.334469 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:12.334476 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:12.335267 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:12.335270 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:12.336078 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:12.336087 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:12.336750 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:12.336754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:12.336839 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:12.336856 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 12:46:12.336924 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 12:46:12.336989 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":72,"spinlock_wait_cycles":7040,"thread_start_us":76,"threads_started":1}
I20260927 12:46:12.337153 2063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:12.333516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38285 (local address 127.0.98.1:49803)
0927 12:46:12.333865 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:12.333876 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:12.333948 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:12.334230 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:12.334232 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:12.334266 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:46:12.334469 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:12.334476 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:12.335258 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:12.335262 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:12.336071 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:12.336081 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:12.336708 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:12.336712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:12.336804 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:12.336828 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 12:46:12.336887 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 12:46:12.336959 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":42,"spinlock_wait_cycles":26496,"thread_start_us":78,"threads_started":1}
I20260927 12:46:12.337585 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "893a0b4abfbf4ba8bcbdea8c44816ab2" candidate_uuid: "4fff3a5975fc41b488a24500cb8a3e42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58e5f2ac64854c16bef4af9c707f5074" is_pre_election: true
I20260927 12:46:12.337755 1947 raft_consensus.cc:2454] T 893a0b4abfbf4ba8bcbdea8c44816ab2 P 58e5f2ac64854c16bef4af9c707f5074 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fff3a5975fc41b488a24500cb8a3e42 in term 0.
I20260927 12:46:12.337994 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "893a0b4abfbf4ba8bcbdea8c44816ab2" candidate_uuid: "4fff3a5975fc41b488a24500cb8a3e42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae350714d3e54c5697b0b8e1c83ac090" is_pre_election: true
I20260927 12:46:12.338116 1598 leader_election.cc:302] T 893a0b4abfbf4ba8bcbdea8c44816ab2 P 4fff3a5975fc41b488a24500cb8a3e42 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fff3a5975fc41b488a24500cb8a3e42, 58e5f2ac64854c16bef4af9c707f5074; no voters:
I20260927 12:46:12.338248 1807 raft_consensus.cc:2454] T 893a0b4abfbf4ba8bcbdea8c44816ab2 P ae350714d3e54c5697b0b8e1c83ac090 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fff3a5975fc41b488a24500cb8a3e42 in term 0.
I20260927 12:46:12.338373 2024 raft_consensus.cc:2781] T 893a0b4abfbf4ba8bcbdea8c44816ab2 P 4fff3a5975fc41b488a24500cb8a3e42 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:46:12.338429 2024 raft_consensus.cc:482] T 893a0b4abfbf4ba8bcbdea8c44816ab2 P 4fff3a5975fc41b488a24500cb8a3e42 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:46:12.338466 2024 raft_consensus.cc:3037] T 893a0b4abfbf4ba8bcbdea8c44816ab2 P 4fff3a5975fc41b488a24500cb8a3e42 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:46:12.339519 2024 raft_consensus.cc:504] T 893a0b4abfbf4ba8bcbdea8c44816ab2 P 4fff3a5975fc41b488a24500cb8a3e42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fff3a5975fc41b488a24500cb8a3e42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34355 } } peers { permanent_uuid: "ae350714d3e54c5697b0b8e1c83ac090" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38285 } } peers { permanent_uuid: "58e5f2ac64854c16bef4af9c707f5074" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44099 } }
I20260927 12:46:12.339861 2024 leader_election.cc:288] T 893a0b4abfbf4ba8bcbdea8c44816ab2 P 4fff3a5975fc41b488a24500cb8a3e42 [CANDIDATE]: Term 1 election: Requested vote from peers ae350714d3e54c5697b0b8e1c83ac090 (127.0.98.2:38285), 58e5f2ac64854c16bef4af9c707f5074 (127.0.98.3:44099)
I20260927 12:46:12.340147 1807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "893a0b4abfbf4ba8bcbdea8c44816ab2" candidate_uuid: "4fff3a5975fc41b488a24500cb8a3e42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae350714d3e54c5697b0b8e1c83ac090"
I20260927 12:46:12.340153 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "893a0b4abfbf4ba8bcbdea8c44816ab2" candidate_uuid: "4fff3a5975fc41b488a24500cb8a3e42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58e5f2ac64854c16bef4af9c707f5074"
I20260927 12:46:12.340263 1947 raft_consensus.cc:3037] T 893a0b4abfbf4ba8bcbdea8c44816ab2 P 58e5f2ac64854c16bef4af9c707f5074 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:46:12.340263 1807 raft_consensus.cc:3037] T 893a0b4abfbf4ba8bcbdea8c44816ab2 P ae350714d3e54c5697b0b8e1c83ac090 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:46:12.341346 1807 raft_consensus.cc:2454] T 893a0b4abfbf4ba8bcbdea8c44816ab2 P ae350714d3e54c5697b0b8e1c83ac090 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fff3a5975fc41b488a24500cb8a3e42 in term 1.
I20260927 12:46:12.341346 1947 raft_consensus.cc:2454] T 893a0b4abfbf4ba8bcbdea8c44816ab2 P 58e5f2ac64854c16bef4af9c707f5074 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fff3a5975fc41b488a24500cb8a3e42 in term 1.
I20260927 12:46:12.341601 1598 leader_election.cc:302] T 893a0b4abfbf4ba8bcbdea8c44816ab2 P 4fff3a5975fc41b488a24500cb8a3e42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fff3a5975fc41b488a24500cb8a3e42, 58e5f2ac64854c16bef4af9c707f5074; no voters:
I20260927 12:46:12.341744 2024 raft_consensus.cc:2781] T 893a0b4abfbf4ba8bcbdea8c44816ab2 P 4fff3a5975fc41b488a24500cb8a3e42 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:46:12.341948 2024 raft_consensus.cc:686] T 893a0b4abfbf4ba8bcbdea8c44816ab2 P 4fff3a5975fc41b488a24500cb8a3e42 [term 1 LEADER]: Becoming Leader. State: Replica: 4fff3a5975fc41b488a24500cb8a3e42, State: Running, Role: LEADER
I20260927 12:46:12.342201 2024 consensus_queue.cc:237] T 893a0b4abfbf4ba8bcbdea8c44816ab2 P 4fff3a5975fc41b488a24500cb8a3e42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fff3a5975fc41b488a24500cb8a3e42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34355 } } peers { permanent_uuid: "ae350714d3e54c5697b0b8e1c83ac090" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38285 } } peers { permanent_uuid: "58e5f2ac64854c16bef4af9c707f5074" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44099 } }
I20260927 12:46:12.343251 2067 tablet_replica.cc:439] T 893a0b4abfbf4ba8bcbdea8c44816ab2 P 4fff3a5975fc41b488a24500cb8a3e42: TxnStatusTablet state changed. Reason: New leader 4fff3a5975fc41b488a24500cb8a3e42. Latest consensus state: current_term: 1 leader_uuid: "4fff3a5975fc41b488a24500cb8a3e42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fff3a5975fc41b488a24500cb8a3e42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34355 } } peers { permanent_uuid: "ae350714d3e54c5697b0b8e1c83ac090" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38285 } } peers { permanent_uuid: "58e5f2ac64854c16bef4af9c707f5074" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44099 } } }
I20260927 12:46:12.343358 2067 tablet_replica.cc:442] T 893a0b4abfbf4ba8bcbdea8c44816ab2 P 4fff3a5975fc41b488a24500cb8a3e42: This TxnStatusTablet replica's current role is: LEADER
I20260927 12:46:12.343627 2069 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 12:46:12.343832 2069 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 12:46:12.344271 1509 catalog_manager.cc:5495] T 893a0b4abfbf4ba8bcbdea8c44816ab2 P 4fff3a5975fc41b488a24500cb8a3e42 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4fff3a5975fc41b488a24500cb8a3e42 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "4fff3a5975fc41b488a24500cb8a3e42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fff3a5975fc41b488a24500cb8a3e42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae350714d3e54c5697b0b8e1c83ac090" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58e5f2ac64854c16bef4af9c707f5074" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44099 } health_report { overall_health: UNKNOWN } } }
I20260927 12:46:12.403781 2072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:12.400098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51570 (local address 127.0.98.1:34355)
0927 12:46:12.400435 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:12.400448 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:12.400464 (+ 16us) server_negotiation.cc:407] Connection header received
0927 12:46:12.400531 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:12.400533 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:12.400585 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:12.400676 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:46:12.400903 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:12.401424 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:12.402281 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:12.402540 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:12.402623 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:12.402911 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:46:12.403374 (+ 463us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:46:12.403475 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:12.403564 (+ 89us) server_negotiation.cc:299] Negotiation successful
0927 12:46:12.403615 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":160,"threads_started":1}
I20260927 12:46:12.403770 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:12.399985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34355 (local address 127.0.0.1:51570)
0927 12:46:12.400228 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:12.400240 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:12.400369 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:12.400703 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:12.400706 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:12.400713 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 12:46:12.400812 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:12.400818 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:12.401520 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:12.401524 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:12.402175 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:12.402184 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:12.402636 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:12.402639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:12.402734 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:12.402776 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 12:46:12.403460 (+ 684us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 12:46:12.403495 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 12:46:12.403554 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 12:46:12.403642 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152}
I20260927 12:46:12.406760 1947 raft_consensus.cc:1261] T 893a0b4abfbf4ba8bcbdea8c44816ab2 P 58e5f2ac64854c16bef4af9c707f5074 [term 1 FOLLOWER]: Refusing update from remote peer 4fff3a5975fc41b488a24500cb8a3e42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:46:12.407219 2067 consensus_queue.cc:1035] T 893a0b4abfbf4ba8bcbdea8c44816ab2 P 4fff3a5975fc41b488a24500cb8a3e42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58e5f2ac64854c16bef4af9c707f5074" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:46:12.407383 1807 raft_consensus.cc:1261] T 893a0b4abfbf4ba8bcbdea8c44816ab2 P ae350714d3e54c5697b0b8e1c83ac090 [term 1 FOLLOWER]: Refusing update from remote peer 4fff3a5975fc41b488a24500cb8a3e42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:46:12.407867 2067 consensus_queue.cc:1035] T 893a0b4abfbf4ba8bcbdea8c44816ab2 P 4fff3a5975fc41b488a24500cb8a3e42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae350714d3e54c5697b0b8e1c83ac090" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:46:12.408562 2022 tablet_replica.cc:439] T 893a0b4abfbf4ba8bcbdea8c44816ab2 P 58e5f2ac64854c16bef4af9c707f5074: TxnStatusTablet state changed. Reason: New leader 4fff3a5975fc41b488a24500cb8a3e42. Latest consensus state: current_term: 1 leader_uuid: "4fff3a5975fc41b488a24500cb8a3e42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fff3a5975fc41b488a24500cb8a3e42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34355 } } peers { permanent_uuid: "ae350714d3e54c5697b0b8e1c83ac090" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38285 } } peers { permanent_uuid: "58e5f2ac64854c16bef4af9c707f5074" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44099 } } }
I20260927 12:46:12.408675 2022 tablet_replica.cc:442] T 893a0b4abfbf4ba8bcbdea8c44816ab2 P 58e5f2ac64854c16bef4af9c707f5074: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 12:46:12.410501 2027 tablet_replica.cc:439] T 893a0b4abfbf4ba8bcbdea8c44816ab2 P ae350714d3e54c5697b0b8e1c83ac090: TxnStatusTablet state changed. Reason: New leader 4fff3a5975fc41b488a24500cb8a3e42. Latest consensus state: current_term: 1 leader_uuid: "4fff3a5975fc41b488a24500cb8a3e42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fff3a5975fc41b488a24500cb8a3e42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34355 } } peers { permanent_uuid: "ae350714d3e54c5697b0b8e1c83ac090" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38285 } } peers { permanent_uuid: "58e5f2ac64854c16bef4af9c707f5074" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44099 } } }
I20260927 12:46:12.410622 2027 tablet_replica.cc:442] T 893a0b4abfbf4ba8bcbdea8c44816ab2 P ae350714d3e54c5697b0b8e1c83ac090: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 12:46:12.411963 2067 tablet_replica.cc:439] T 893a0b4abfbf4ba8bcbdea8c44816ab2 P 4fff3a5975fc41b488a24500cb8a3e42: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4fff3a5975fc41b488a24500cb8a3e42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fff3a5975fc41b488a24500cb8a3e42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34355 } } peers { permanent_uuid: "ae350714d3e54c5697b0b8e1c83ac090" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38285 } } peers { permanent_uuid: "58e5f2ac64854c16bef4af9c707f5074" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44099 } } }
I20260927 12:46:12.412071 2067 tablet_replica.cc:442] T 893a0b4abfbf4ba8bcbdea8c44816ab2 P 4fff3a5975fc41b488a24500cb8a3e42: This TxnStatusTablet replica's current role is: LEADER
I20260927 12:46:12.412182 2067 tablet_replica.cc:439] T 893a0b4abfbf4ba8bcbdea8c44816ab2 P 4fff3a5975fc41b488a24500cb8a3e42: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4fff3a5975fc41b488a24500cb8a3e42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fff3a5975fc41b488a24500cb8a3e42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34355 } } peers { permanent_uuid: "ae350714d3e54c5697b0b8e1c83ac090" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38285 } } peers { permanent_uuid: "58e5f2ac64854c16bef4af9c707f5074" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44099 } } }
I20260927 12:46:12.412251 2067 tablet_replica.cc:442] T 893a0b4abfbf4ba8bcbdea8c44816ab2 P 4fff3a5975fc41b488a24500cb8a3e42: This TxnStatusTablet replica's current role is: LEADER
I20260927 12:46:12.413156 2058 mvcc.cc:205] Tried to move back new op lower bound from 7333941954172985344 to 7333941953915781120. Current Snapshot: MvccSnapshot[applied={T|T < 7333941954172985344}]
I20260927 12:46:12.413208 2022 tablet_replica.cc:439] T 893a0b4abfbf4ba8bcbdea8c44816ab2 P 58e5f2ac64854c16bef4af9c707f5074: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4fff3a5975fc41b488a24500cb8a3e42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fff3a5975fc41b488a24500cb8a3e42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34355 } } peers { permanent_uuid: "ae350714d3e54c5697b0b8e1c83ac090" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38285 } } peers { permanent_uuid: "58e5f2ac64854c16bef4af9c707f5074" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44099 } } }
I20260927 12:46:12.413314 2022 tablet_replica.cc:442] T 893a0b4abfbf4ba8bcbdea8c44816ab2 P 58e5f2ac64854c16bef4af9c707f5074: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 12:46:12.413480 2027 tablet_replica.cc:439] T 893a0b4abfbf4ba8bcbdea8c44816ab2 P ae350714d3e54c5697b0b8e1c83ac090: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4fff3a5975fc41b488a24500cb8a3e42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fff3a5975fc41b488a24500cb8a3e42" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34355 } } peers { permanent_uuid: "ae350714d3e54c5697b0b8e1c83ac090" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38285 } } peers { permanent_uuid: "58e5f2ac64854c16bef4af9c707f5074" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44099 } } }
I20260927 12:46:12.413574 2027 tablet_replica.cc:442] T 893a0b4abfbf4ba8bcbdea8c44816ab2 P ae350714d3e54c5697b0b8e1c83ac090: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 12:46:12.428105 2072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:12.424757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52581 (local address 127.0.98.1:34355)
0927 12:46:12.424951 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:12.424954 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:12.425042 (+ 88us) server_negotiation.cc:407] Connection header received
0927 12:46:12.425161 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:12.425163 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:12.425205 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:12.425266 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:46:12.425478 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:12.425974 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:12.427145 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:12.427358 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:12.427434 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:12.427727 (+ 293us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:46:12.427798 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:46:12.427851 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:12.427948 (+ 97us) server_negotiation.cc:299] Negotiation successful
0927 12:46:12.427993 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260927 12:46:12.428192 2081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:12.424639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34355 (local address 127.0.98.2:52581)
0927 12:46:12.424957 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:12.424972 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:12.425079 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:12.425283 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:12.425285 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:12.425292 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 12:46:12.425400 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:12.425406 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:12.426081 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:12.426084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:12.427033 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:12.427047 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:12.427461 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:12.427465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:12.427552 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:12.427608 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 12:46:12.427870 (+ 262us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 12:46:12.427876 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 12:46:12.427943 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 12:46:12.428038 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":114,"threads_started":1}
I20260927 12:46:12.455515 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:12.450818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53055 (local address 127.0.98.2:38285)
0927 12:46:12.450924 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:12.450930 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:12.451720 (+ 790us) server_negotiation.cc:407] Connection header received
0927 12:46:12.451959 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:12.451962 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:12.452026 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:12.452121 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:46:12.452403 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:12.453196 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:12.454268 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:12.454571 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:12.454678 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:12.455018 (+ 340us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:46:12.455106 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:46:12.455174 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:12.455309 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 12:46:12.455372 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 12:46:12.455556 2083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:12.450744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38285 (local address 127.0.98.1:53055)
0927 12:46:12.451211 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:12.451243 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:12.451818 (+ 575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:12.452137 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:12.452140 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:12.452149 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 12:46:12.452300 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:12.452308 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:12.453364 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:12.453366 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:12.454150 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:12.454163 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:12.454708 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:12.454712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:12.454837 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:12.454886 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 12:46:12.455201 (+ 315us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 12:46:12.455209 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 12:46:12.455283 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 12:46:12.455386 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":87,"threads_started":1}
I20260927 12:46:12.481043 1508 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:46:12.481195 1508 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:46:12.483336 1508 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f8fd783ac9124faa86455d32e73f8d7a: Sending DeleteTablet for 3 replicas of tablet 99cc3b6d1004452682548ab91f4abfab
I20260927 12:46:12.483951 1647 tablet_service.cc:1514] Processing DeleteTablet for tablet 99cc3b6d1004452682548ab91f4abfab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:46:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35128
I20260927 12:46:12.483959 1787 tablet_service.cc:1514] Processing DeleteTablet for tablet 99cc3b6d1004452682548ab91f4abfab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:46:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38360
I20260927 12:46:12.484020 1927 tablet_service.cc:1514] Processing DeleteTablet for tablet 99cc3b6d1004452682548ab91f4abfab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:46:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37512
I20260927 12:46:12.484349 2086 tablet_replica.cc:330] T 99cc3b6d1004452682548ab91f4abfab P ae350714d3e54c5697b0b8e1c83ac090: stopping tablet replica
I20260927 12:46:12.484475 2086 raft_consensus.cc:2229] T 99cc3b6d1004452682548ab91f4abfab P ae350714d3e54c5697b0b8e1c83ac090 [term 1 LEADER]: Raft consensus shutting down.
I20260927 12:46:12.484508 2085 tablet_replica.cc:330] T 99cc3b6d1004452682548ab91f4abfab P 4fff3a5975fc41b488a24500cb8a3e42: stopping tablet replica
I20260927 12:46:12.484607 2085 raft_consensus.cc:2229] T 99cc3b6d1004452682548ab91f4abfab P 4fff3a5975fc41b488a24500cb8a3e42 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:46:12.484674 2086 raft_consensus.cc:2258] T 99cc3b6d1004452682548ab91f4abfab P ae350714d3e54c5697b0b8e1c83ac090 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:46:12.484702 2085 raft_consensus.cc:2258] T 99cc3b6d1004452682548ab91f4abfab P 4fff3a5975fc41b488a24500cb8a3e42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:46:12.484912 2087 tablet_replica.cc:330] T 99cc3b6d1004452682548ab91f4abfab P 58e5f2ac64854c16bef4af9c707f5074: stopping tablet replica
I20260927 12:46:12.485018 2087 raft_consensus.cc:2229] T 99cc3b6d1004452682548ab91f4abfab P 58e5f2ac64854c16bef4af9c707f5074 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:46:12.485121 2087 raft_consensus.cc:2258] T 99cc3b6d1004452682548ab91f4abfab P 58e5f2ac64854c16bef4af9c707f5074 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:46:12.485610 2085 ts_tablet_manager.cc:1905] T 99cc3b6d1004452682548ab91f4abfab P 4fff3a5975fc41b488a24500cb8a3e42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:46:12.485654 2087 ts_tablet_manager.cc:1905] T 99cc3b6d1004452682548ab91f4abfab P 58e5f2ac64854c16bef4af9c707f5074: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:46:12.486687 2086 ts_tablet_manager.cc:1905] T 99cc3b6d1004452682548ab91f4abfab P ae350714d3e54c5697b0b8e1c83ac090: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:46:12.488907 2085 ts_tablet_manager.cc:1918] T 99cc3b6d1004452682548ab91f4abfab P 4fff3a5975fc41b488a24500cb8a3e42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 12:46:12.488996 2085 log.cc:1196] T 99cc3b6d1004452682548ab91f4abfab P 4fff3a5975fc41b488a24500cb8a3e42: Deleting WAL directory at /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/minicluster-data/ts-0/wal/wals/99cc3b6d1004452682548ab91f4abfab
I20260927 12:46:12.489282 2085 ts_tablet_manager.cc:1939] T 99cc3b6d1004452682548ab91f4abfab P 4fff3a5975fc41b488a24500cb8a3e42: Deleting consensus metadata
I20260927 12:46:12.489770 2086 ts_tablet_manager.cc:1918] T 99cc3b6d1004452682548ab91f4abfab P ae350714d3e54c5697b0b8e1c83ac090: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 12:46:12.489852 2086 log.cc:1196] T 99cc3b6d1004452682548ab91f4abfab P ae350714d3e54c5697b0b8e1c83ac090: Deleting WAL directory at /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/minicluster-data/ts-1/wal/wals/99cc3b6d1004452682548ab91f4abfab
I20260927 12:46:12.490125 2086 ts_tablet_manager.cc:1939] T 99cc3b6d1004452682548ab91f4abfab P ae350714d3e54c5697b0b8e1c83ac090: Deleting consensus metadata
I20260927 12:46:12.490128 1496 catalog_manager.cc:4841] TS 4fff3a5975fc41b488a24500cb8a3e42 (127.0.98.1:34355): tablet 99cc3b6d1004452682548ab91f4abfab (table test-table [id=1a8aff8e653342fc9dde2fd82dde2c0b]) successfully deleted
I20260927 12:46:12.490852 1495 catalog_manager.cc:4841] TS ae350714d3e54c5697b0b8e1c83ac090 (127.0.98.2:38285): tablet 99cc3b6d1004452682548ab91f4abfab (table test-table [id=1a8aff8e653342fc9dde2fd82dde2c0b]) successfully deleted
I20260927 12:46:12.492385 2087 ts_tablet_manager.cc:1918] T 99cc3b6d1004452682548ab91f4abfab P 58e5f2ac64854c16bef4af9c707f5074: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 12:46:12.492473 2087 log.cc:1196] T 99cc3b6d1004452682548ab91f4abfab P 58e5f2ac64854c16bef4af9c707f5074: Deleting WAL directory at /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790513161160093-392-0/minicluster-data/ts-2/wal/wals/99cc3b6d1004452682548ab91f4abfab
I20260927 12:46:12.492803 2087 ts_tablet_manager.cc:1939] T 99cc3b6d1004452682548ab91f4abfab P 58e5f2ac64854c16bef4af9c707f5074: Deleting consensus metadata
I20260927 12:46:12.493537 1494 catalog_manager.cc:4841] TS 58e5f2ac64854c16bef4af9c707f5074 (127.0.98.3:44099): tablet 99cc3b6d1004452682548ab91f4abfab (table test-table [id=1a8aff8e653342fc9dde2fd82dde2c0b]) successfully deleted
Sep 27 12:46:12 dist-test-slave-lkn4 krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513171, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 12:46:12.507474 2072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:12.486032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51586 (local address 127.0.98.1:34355)
0927 12:46:12.486166 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:12.486170 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:12.493699 (+ 7529us) server_negotiation.cc:407] Connection header received
0927 12:46:12.493846 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:12.493848 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:12.493895 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:12.493958 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:12.494874 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:12.495425 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:12.496866 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:12.497084 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:12.497160 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:12.503753 (+ 6593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:12.503764 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:12.503778 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:12.503827 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:12.505513 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:12.506120 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:12.506130 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:12.506131 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:12.506225 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:12.506519 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:12.506522 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:12.506524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:12.506796 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:12.506980 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:12.507198 (+ 218us) server_negotiation.cc:299] Negotiation successful
0927 12:46:12.507311 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260927 12:46:12.507900 1699 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51586
I20260927 12:46:12.516055 2092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:12.509497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58406 (local address 127.0.98.62:46513)
0927 12:46:12.509765 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:12.509769 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:12.509780 (+ 11us) server_negotiation.cc:407] Connection header received
0927 12:46:12.509816 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:12.509818 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:12.509858 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:12.509930 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:12.510511 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:12.511093 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:12.511797 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:12.512017 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:12.512091 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:12.513350 (+ 1259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:12.513352 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:12.513354 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:12.513377 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:12.514786 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:12.515122 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:12.515124 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:12.515125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:12.515193 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:12.515370 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:12.515372 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:12.515373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:12.515528 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:12.515622 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:12.515788 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 12:46:12.515885 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":74,"threads_started":1}
W20260927 12:46:12.516424 1508 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58406
I20260927 12:46:12.517482 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 1582
I20260927 12:46:12.523892 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 1722
I20260927 12:46:12.530393 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 1862
I20260927 12:46:12.535563 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 1478
2026-09-27T12:46:12Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5075 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:46:12 dist-test-slave-lkn4 krb5kdc[2095](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:46:12 dist-test-slave-lkn4 krb5kdc[2095](info): set up 2 sockets
Sep 27 12:46:12 dist-test-slave-lkn4 krb5kdc[2095](info): commencing operation
krb5kdc: starting...
W20260927 12:46:13.493641 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/krb5kdc/test-user.keytab.
Sep 27 12:46:13 dist-test-slave-lkn4 krb5kdc[2095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:46:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:46:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:46:13.613860 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41189
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41189
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:46:13.724027 2108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:46:13.724324 2108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:46:13.724385 2108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:46:13.728471 2108 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:46:13.728536 2108 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 12:46:13.728559 2108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:46:13.728581 2108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:46:13.728603 2108 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:46:13.728631 2108 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:46:13.733989 2108 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41189
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:46:13.735257 2108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:46:13.736426 2108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:46:13.744786 2113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:13.744788 2116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:13.744908 2108 server_base.cc:1034] running on GCE node
W20260927 12:46:13.744786 2114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:13.745568 2108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:46:13.746757 2108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:46:13.747917 2108 hybrid_clock.cc:648] HybridClock initialized: now 1790513173747904 us; error 29 us; skew 500 ppm
Sep 27 12:46:13 dist-test-slave-lkn4 krb5kdc[2095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513173, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:46:13.750571 2108 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:46:13.751447 2108 webserver.cc:460] Webserver started at http://127.0.98.62:38799/ using document root <none> and password file <none>
I20260927 12:46:13.751966 2108 fs_manager.cc:359] Metadata directory not provided
I20260927 12:46:13.752027 2108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:46:13.752213 2108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:46:13.754259 2108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/master-0/data/instance:
uuid: "92ba4aec5bf34fde9f6049baedcb6f3f"
format_stamp: "Formatted at 2026-09-27 12:46:13 on dist-test-slave-lkn4"
server_key: "00b876e31a7ad56170f0da70a6d698f7"
server_key_iv: "9db764ccfb5c87528640727b3ae8ea92"
server_key_version: "encryptionkey@0"
I20260927 12:46:13.754760 2108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/master-0/wal/instance:
uuid: "92ba4aec5bf34fde9f6049baedcb6f3f"
format_stamp: "Formatted at 2026-09-27 12:46:13 on dist-test-slave-lkn4"
server_key: "00b876e31a7ad56170f0da70a6d698f7"
server_key_iv: "9db764ccfb5c87528640727b3ae8ea92"
server_key_version: "encryptionkey@0"
I20260927 12:46:13.758332 2108 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:46:13.760612 2123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:13.761818 2108 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:46:13.761920 2108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/master-0/wal
uuid: "92ba4aec5bf34fde9f6049baedcb6f3f"
format_stamp: "Formatted at 2026-09-27 12:46:13 on dist-test-slave-lkn4"
server_key: "00b876e31a7ad56170f0da70a6d698f7"
server_key_iv: "9db764ccfb5c87528640727b3ae8ea92"
server_key_version: "encryptionkey@0"
I20260927 12:46:13.762023 2108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:46:13.778016 2108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:46:13.781351 2108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:46:13.781529 2108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:46:13.789366 2108 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41189
I20260927 12:46:13.789390 2185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41189 every 8 connection(s)
I20260927 12:46:13.790383 2108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:46:13.793202 2186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:46:13.798940 2186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92ba4aec5bf34fde9f6049baedcb6f3f: Bootstrap starting.
I20260927 12:46:13.800285 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 2108
I20260927 12:46:13.800381 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/master-0/wal/instance
I20260927 12:46:13.800635 392 external_mini_cluster.cc:1428] Setting key 2a925cc93050ff4b5adaf05a8cfcb2dd
I20260927 12:46:13.801529 2186 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92ba4aec5bf34fde9f6049baedcb6f3f: Neither blocks nor log segments found. Creating new log.
I20260927 12:46:13.802441 2186 log.cc:824] T 00000000000000000000000000000000 P 92ba4aec5bf34fde9f6049baedcb6f3f: Log is configured to *not* fsync() on all Append() calls
I20260927 12:46:13.804289 2186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92ba4aec5bf34fde9f6049baedcb6f3f: No bootstrap required, opened a new log
I20260927 12:46:13.807698 2186 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92ba4aec5bf34fde9f6049baedcb6f3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92ba4aec5bf34fde9f6049baedcb6f3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41189 } }
I20260927 12:46:13.807947 2186 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92ba4aec5bf34fde9f6049baedcb6f3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:46:13.808012 2186 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92ba4aec5bf34fde9f6049baedcb6f3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92ba4aec5bf34fde9f6049baedcb6f3f, State: Initialized, Role: FOLLOWER
I20260927 12:46:13.808432 2186 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92ba4aec5bf34fde9f6049baedcb6f3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92ba4aec5bf34fde9f6049baedcb6f3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41189 } }
I20260927 12:46:13.808557 2186 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92ba4aec5bf34fde9f6049baedcb6f3f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:46:13.808622 2186 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92ba4aec5bf34fde9f6049baedcb6f3f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:46:13.808720 2186 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92ba4aec5bf34fde9f6049baedcb6f3f [term 0 FOLLOWER]: Advancing to term 1
Sep 27 12:46:13 dist-test-slave-lkn4 krb5kdc[2095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:46:13.809746 2186 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92ba4aec5bf34fde9f6049baedcb6f3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92ba4aec5bf34fde9f6049baedcb6f3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41189 } }
I20260927 12:46:13.810102 2186 leader_election.cc:302] T 00000000000000000000000000000000 P 92ba4aec5bf34fde9f6049baedcb6f3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92ba4aec5bf34fde9f6049baedcb6f3f; no voters:
I20260927 12:46:13.810429 2186 leader_election.cc:288] T 00000000000000000000000000000000 P 92ba4aec5bf34fde9f6049baedcb6f3f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:46:13.810519 2191 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92ba4aec5bf34fde9f6049baedcb6f3f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:46:13.810714 2191 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92ba4aec5bf34fde9f6049baedcb6f3f [term 1 LEADER]: Becoming Leader. State: Replica: 92ba4aec5bf34fde9f6049baedcb6f3f, State: Running, Role: LEADER
I20260927 12:46:13.811029 2191 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92ba4aec5bf34fde9f6049baedcb6f3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92ba4aec5bf34fde9f6049baedcb6f3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41189 } }
I20260927 12:46:13.811787 2186 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92ba4aec5bf34fde9f6049baedcb6f3f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:46:13.812103 2193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92ba4aec5bf34fde9f6049baedcb6f3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92ba4aec5bf34fde9f6049baedcb6f3f. Latest consensus state: current_term: 1 leader_uuid: "92ba4aec5bf34fde9f6049baedcb6f3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92ba4aec5bf34fde9f6049baedcb6f3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41189 } } }
I20260927 12:46:13.812217 2193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92ba4aec5bf34fde9f6049baedcb6f3f [sys.catalog]: This master's current role is: LEADER
I20260927 12:46:13.812580 2192 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92ba4aec5bf34fde9f6049baedcb6f3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "92ba4aec5bf34fde9f6049baedcb6f3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92ba4aec5bf34fde9f6049baedcb6f3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41189 } } }
I20260927 12:46:13.812680 2192 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92ba4aec5bf34fde9f6049baedcb6f3f [sys.catalog]: This master's current role is: LEADER
I20260927 12:46:13.812947 2195 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:46:13.815515 2189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:13.801830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55230 (local address 127.0.98.62:41189)
0927 12:46:13.802333 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:13.802349 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:13.802387 (+ 38us) server_negotiation.cc:407] Connection header received
0927 12:46:13.803087 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:13.803108 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:13.803462 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:13.803787 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:13.804564 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:13.805572 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:13.806477 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:13.806988 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:13.807138 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:13.809707 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:13.809712 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:13.809727 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:13.809757 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:13.811809 (+ 2052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:13.812461 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:13.812468 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:13.812475 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:13.812626 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:13.813070 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:13.813074 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:13.813075 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:13.813566 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:13.813737 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:13.814205 (+ 468us) server_negotiation.cc:299] Negotiation successful
0927 12:46:13.814525 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":169,"threads_started":1}
I20260927 12:46:13.816733 2195 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:46:13.823031 2195 catalog_manager.cc:1347] Generated new cluster ID: ff008e5993b64ad7bc0879e46548d674
I20260927 12:46:13.823120 2195 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:46:13.847673 2195 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:46:13.849042 2195 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:46:13.861338 2195 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 92ba4aec5bf34fde9f6049baedcb6f3f: Generated new TSK 0
I20260927 12:46:13.862301 2195 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 12:46:13.893599 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41189
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:46:14.008921 2212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:46:14.009205 2212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:46:14.009274 2212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:46:14.013548 2212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:46:14.013621 2212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:46:14.013679 2212 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 12:46:14.013707 2212 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 12:46:14.013761 2212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:46:14.019309 2212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:41189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:46:14.020427 2212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:46:14.021457 2212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:46:14.031167 2217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:14.031203 2212 server_base.cc:1034] running on GCE node
W20260927 12:46:14.031216 2218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:14.031167 2220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:14.031778 2212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:46:14.032445 2212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:46:14.033601 2212 hybrid_clock.cc:648] HybridClock initialized: now 1790513174033579 us; error 40 us; skew 500 ppm
Sep 27 12:46:14 dist-test-slave-lkn4 krb5kdc[2095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513174, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:46:14.036439 2212 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 12:46:14.037664 2212 webserver.cc:460] Webserver started at http://127.0.98.1:41839/ using document root <none> and password file <none>
I20260927 12:46:14.038316 2212 fs_manager.cc:359] Metadata directory not provided
I20260927 12:46:14.038400 2212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:46:14.038618 2212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:46:14.040764 2212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/ts-0/data/instance:
uuid: "c60eee98017b49d4aa748b9ac4846cde"
format_stamp: "Formatted at 2026-09-27 12:46:14 on dist-test-slave-lkn4"
server_key: "aaa168c88d60b6ff89bfa148d9d29976"
server_key_iv: "c5379faf8801ca3a08198653a645de1a"
server_key_version: "encryptionkey@0"
I20260927 12:46:14.041320 2212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c60eee98017b49d4aa748b9ac4846cde"
format_stamp: "Formatted at 2026-09-27 12:46:14 on dist-test-slave-lkn4"
server_key: "aaa168c88d60b6ff89bfa148d9d29976"
server_key_iv: "c5379faf8801ca3a08198653a645de1a"
server_key_version: "encryptionkey@0"
I20260927 12:46:14.045288 2212 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 12:46:14.047932 2227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:14.049216 2212 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:46:14.049331 2212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/ts-0/wal
uuid: "c60eee98017b49d4aa748b9ac4846cde"
format_stamp: "Formatted at 2026-09-27 12:46:14 on dist-test-slave-lkn4"
server_key: "aaa168c88d60b6ff89bfa148d9d29976"
server_key_iv: "c5379faf8801ca3a08198653a645de1a"
server_key_version: "encryptionkey@0"
I20260927 12:46:14.049437 2212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:46:14.069924 2212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:46:14.073081 2212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:46:14.073282 2212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:46:14.074898 2212 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:46:14.074965 2212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:14.075024 2212 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:46:14.075049 2212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 12:46:14 dist-test-slave-lkn4 krb5kdc[2095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513174, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:46:14.085386 2212 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37687
I20260927 12:46:14.085485 2342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37687 every 8 connection(s)
I20260927 12:46:14.086390 2212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:46:14.089159 2189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:14.075730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40069 (local address 127.0.98.62:41189)
0927 12:46:14.075878 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:14.075883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:14.076877 (+ 994us) server_negotiation.cc:407] Connection header received
0927 12:46:14.078152 (+ 1275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:14.078156 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:14.078218 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:14.078324 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:14.080010 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:14.080719 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:14.082093 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:14.082358 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:14.082469 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:14.085440 (+ 2971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:14.085444 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:14.085447 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:14.085481 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:14.087156 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:14.087648 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:14.087651 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:14.087652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:14.087739 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:14.088040 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:14.088042 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:14.088043 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:14.088220 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:14.088309 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:14.088897 (+ 588us) server_negotiation.cc:299] Negotiation successful
0927 12:46:14.089031 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 12:46:14.090113 2236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:14.076182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41189 (local address 127.0.98.1:40069)
0927 12:46:14.076689 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:14.076751 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:14.077870 (+ 1119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:14.078535 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:14.078545 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:14.079056 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:14.079820 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:14.079842 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:14.080895 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:14.080898 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:14.081945 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:14.081956 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:14.082526 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:14.082530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:14.082737 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:14.083383 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:14.083405 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:14.085232 (+ 1827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:14.087259 (+ 2027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:14.087265 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:14.087281 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:14.087541 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:14.087831 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:14.087833 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:14.087834 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:14.087972 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:14.088346 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:14.088352 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:14.088605 (+ 253us) client_negotiation.cc:769] Sending connection context
0927 12:46:14.088838 (+ 233us) client_negotiation.cc:240] Negotiation successful
0927 12:46:14.089119 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":117,"threads_started":1}
I20260927 12:46:14.090322 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 2212
I20260927 12:46:14.090430 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:46:14.090680 392 external_mini_cluster.cc:1428] Setting key 808b42e2a74a9cd5a3958b62f3f8b35c
I20260927 12:46:14.091584 2343 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41189
I20260927 12:46:14.091753 2343 heartbeater.cc:464] Registering TS with master...
I20260927 12:46:14.092399 2343 heartbeater.cc:511] Master 127.0.98.62:41189 requested a full tablet report, sending...
I20260927 12:46:14.094329 2139 ts_manager.cc:194] Registered new tserver with Master: c60eee98017b49d4aa748b9ac4846cde (127.0.98.1:37687)
I20260927 12:46:14.095762 2139 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40069
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 12:46:14.105278 2350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:14.096338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41189 (local address 127.0.98.1:48631)
0927 12:46:14.098237 (+ 1899us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:14.098253 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:14.098371 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:14.098612 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:14.098615 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:14.098839 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:14.098966 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:14.098974 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:14.099808 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:14.099812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:14.100570 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:14.100580 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:14.101062 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:14.101064 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:14.101128 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:14.101609 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:14.101621 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:14.102032 (+ 411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:14.103836 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:14.103840 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:14.103842 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:14.104101 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:14.104398 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:14.104400 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:14.104401 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:14.104475 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:14.104837 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:14.104839 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:14.104920 (+ 81us) client_negotiation.cc:769] Sending connection context
0927 12:46:14.104994 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 12:46:14.105115 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1799,"thread_start_us":100,"threads_started":1}
I20260927 12:46:14.105356 2189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:14.096671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48631 (local address 127.0.98.62:41189)
0927 12:46:14.096839 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:14.096842 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:14.098335 (+ 1493us) server_negotiation.cc:407] Connection header received
0927 12:46:14.098461 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:14.098463 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:14.098504 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:14.098571 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:14.099091 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:14.099619 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:14.100691 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:14.100957 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:14.101061 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:14.102191 (+ 1130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:14.102194 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:14.102196 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:14.102216 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:14.103682 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:14.104210 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:14.104214 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:14.104216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:14.104315 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:14.104543 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:14.104546 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:14.104548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:14.104761 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:14.104853 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:14.105059 (+ 206us) server_negotiation.cc:299] Negotiation successful
0927 12:46:14.105180 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 12:46:14.119079 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41189
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:46:14.228933 2352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:46:14.229240 2352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:46:14.229310 2352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:46:14.233518 2352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:46:14.233603 2352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:46:14.233664 2352 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 12:46:14.233691 2352 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 12:46:14.233747 2352 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:46:14.239332 2352 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:41189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2352
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:46:14.240430 2352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:46:14.241472 2352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:46:14.251134 2357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:14.251116 2358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:14.251120 2360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:14.251617 2352 server_base.cc:1034] running on GCE node
I20260927 12:46:14.252084 2352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:46:14.252771 2352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:46:14.253932 2352 hybrid_clock.cc:648] HybridClock initialized: now 1790513174253912 us; error 36 us; skew 500 ppm
Sep 27 12:46:14 dist-test-slave-lkn4 krb5kdc[2095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513174, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:46:14.256672 2352 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 12:46:14.257531 2352 webserver.cc:460] Webserver started at http://127.0.98.2:42789/ using document root <none> and password file <none>
I20260927 12:46:14.258103 2352 fs_manager.cc:359] Metadata directory not provided
I20260927 12:46:14.258167 2352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:46:14.258348 2352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:46:14.260337 2352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/ts-1/data/instance:
uuid: "1f0060ce040449f1a4415d377932fa44"
format_stamp: "Formatted at 2026-09-27 12:46:14 on dist-test-slave-lkn4"
server_key: "354230ee2ab77fa59720c961025d4713"
server_key_iv: "32d65568df46a20d7629ae75be97e6ab"
server_key_version: "encryptionkey@0"
I20260927 12:46:14.260843 2352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1f0060ce040449f1a4415d377932fa44"
format_stamp: "Formatted at 2026-09-27 12:46:14 on dist-test-slave-lkn4"
server_key: "354230ee2ab77fa59720c961025d4713"
server_key_iv: "32d65568df46a20d7629ae75be97e6ab"
server_key_version: "encryptionkey@0"
I20260927 12:46:14.264470 2352 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 12:46:14.267124 2367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:14.268388 2352 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:46:14.268482 2352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/ts-1/wal
uuid: "1f0060ce040449f1a4415d377932fa44"
format_stamp: "Formatted at 2026-09-27 12:46:14 on dist-test-slave-lkn4"
server_key: "354230ee2ab77fa59720c961025d4713"
server_key_iv: "32d65568df46a20d7629ae75be97e6ab"
server_key_version: "encryptionkey@0"
I20260927 12:46:14.268569 2352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:46:14.298126 2352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:46:14.301430 2352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:46:14.301594 2352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:46:14.303047 2352 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:46:14.303107 2352 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:14.303162 2352 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:46:14.303192 2352 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 12:46:14 dist-test-slave-lkn4 krb5kdc[2095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513174, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:46:14.313484 2352 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45249
I20260927 12:46:14.313628 2482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45249 every 8 connection(s)
I20260927 12:46:14.314518 2352 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:46:14.315351 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 2352
I20260927 12:46:14.315449 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:46:14.315733 392 external_mini_cluster.cc:1428] Setting key 1f681ac4009d558fbd0ae34b28776d39
I20260927 12:46:14.316656 2189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:14.304064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54055 (local address 127.0.98.62:41189)
0927 12:46:14.304174 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:14.304178 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:14.305346 (+ 1168us) server_negotiation.cc:407] Connection header received
0927 12:46:14.306536 (+ 1190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:14.306540 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:14.306597 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:14.306671 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:14.307947 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:14.308569 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:14.309815 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:14.310086 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:14.310181 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:14.312981 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:14.312985 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:14.312988 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:14.313015 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:14.314493 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:14.315029 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:14.315033 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:14.315035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:14.315122 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:14.315522 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:14.315525 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:14.315527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:14.315756 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:14.315854 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:14.316387 (+ 533us) server_negotiation.cc:299] Negotiation successful
0927 12:46:14.316508 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 12:46:14.317638 2376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:14.304669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41189 (local address 127.0.98.2:54055)
0927 12:46:14.305191 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:14.305236 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:14.306280 (+ 1044us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:14.306865 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:14.306873 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:14.307230 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:14.307797 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:14.307814 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:14.308684 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:14.308688 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:14.309684 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:14.309694 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:14.310245 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:14.310249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:14.310431 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:14.311020 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:14.311037 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:14.312814 (+ 1777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:14.314601 (+ 1787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:14.314607 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:14.314624 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:14.314912 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:14.315221 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:14.315224 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:14.315226 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:14.315407 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:14.315891 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:14.315897 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:14.316152 (+ 255us) client_negotiation.cc:769] Sending connection context
0927 12:46:14.316346 (+ 194us) client_negotiation.cc:240] Negotiation successful
0927 12:46:14.316653 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":147,"threads_started":1}
I20260927 12:46:14.320613 2483 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41189
I20260927 12:46:14.320827 2483 heartbeater.cc:464] Registering TS with master...
I20260927 12:46:14.321448 2483 heartbeater.cc:511] Master 127.0.98.62:41189 requested a full tablet report, sending...
I20260927 12:46:14.322510 2140 ts_manager.cc:194] Registered new tserver with Master: 1f0060ce040449f1a4415d377932fa44 (127.0.98.2:45249)
I20260927 12:46:14.323184 2140 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54055
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 12:46:14.329231 2490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:14.321475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41189 (local address 127.0.98.2:53605)
0927 12:46:14.321730 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:14.321746 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:14.321860 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:14.322224 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:14.322226 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:14.322439 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:14.322563 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:14.322572 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:14.323727 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:14.323730 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:14.324561 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:14.324571 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:14.325133 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:14.325137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:14.325223 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:14.325712 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:14.325726 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:14.326199 (+ 473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:14.327684 (+ 1485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:14.327687 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:14.327688 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:14.327947 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:14.328215 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:14.328218 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:14.328220 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:14.328311 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:14.328792 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:14.328795 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:14.328876 (+ 81us) client_negotiation.cc:769] Sending connection context
0927 12:46:14.328957 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 12:46:14.329073 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":96,"threads_started":1}
I20260927 12:46:14.329435 2189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:14.321097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53605 (local address 127.0.98.62:41189)
0927 12:46:14.321252 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:14.321256 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:14.321838 (+ 582us) server_negotiation.cc:407] Connection header received
0927 12:46:14.321949 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:14.321952 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:14.322005 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:14.322117 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:14.322690 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:14.323602 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:14.324687 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:14.325013 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:14.325118 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:14.326374 (+ 1256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:14.326379 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:14.326382 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:14.326408 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:14.327586 (+ 1178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:14.328046 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:14.328049 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:14.328051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:14.328136 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:14.328388 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:14.328391 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:14.328392 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:14.328663 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:14.328760 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:14.329029 (+ 269us) server_negotiation.cc:299] Negotiation successful
0927 12:46:14.329148 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":53}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 12:46:14.345484 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41189
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:46:14.454144 2492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:46:14.454445 2492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:46:14.454516 2492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:46:14.459003 2492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:46:14.459076 2492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:46:14.459120 2492 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 12:46:14.459136 2492 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 12:46:14.459187 2492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:46:14.464573 2492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:41189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2492
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:46:14.465685 2492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:46:14.466760 2492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:46:14.476286 2497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:14.476300 2500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:14.476300 2498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:14.476649 2492 server_base.cc:1034] running on GCE node
I20260927 12:46:14.477185 2492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:46:14.477808 2492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:46:14.478953 2492 hybrid_clock.cc:648] HybridClock initialized: now 1790513174478925 us; error 41 us; skew 500 ppm
Sep 27 12:46:14 dist-test-slave-lkn4 krb5kdc[2095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513174, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:46:14.481609 2492 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 12:46:14.482597 2492 webserver.cc:460] Webserver started at http://127.0.98.3:42605/ using document root <none> and password file <none>
I20260927 12:46:14.483150 2492 fs_manager.cc:359] Metadata directory not provided
I20260927 12:46:14.483213 2492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:46:14.483414 2492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:46:14.485445 2492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/ts-2/data/instance:
uuid: "2d94505a8b6a4daab8fdbdffdf354596"
format_stamp: "Formatted at 2026-09-27 12:46:14 on dist-test-slave-lkn4"
server_key: "639dd0db7ebd697ebedf94a2dc50c340"
server_key_iv: "da7034656c6dd393ae6d57a8083e3951"
server_key_version: "encryptionkey@0"
I20260927 12:46:14.485957 2492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2d94505a8b6a4daab8fdbdffdf354596"
format_stamp: "Formatted at 2026-09-27 12:46:14 on dist-test-slave-lkn4"
server_key: "639dd0db7ebd697ebedf94a2dc50c340"
server_key_iv: "da7034656c6dd393ae6d57a8083e3951"
server_key_version: "encryptionkey@0"
I20260927 12:46:14.489586 2492 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:46:14.492003 2507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:14.493225 2492 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:46:14.493335 2492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/ts-2/wal
uuid: "2d94505a8b6a4daab8fdbdffdf354596"
format_stamp: "Formatted at 2026-09-27 12:46:14 on dist-test-slave-lkn4"
server_key: "639dd0db7ebd697ebedf94a2dc50c340"
server_key_iv: "da7034656c6dd393ae6d57a8083e3951"
server_key_version: "encryptionkey@0"
I20260927 12:46:14.493422 2492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:46:14.507869 2492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:46:14.510828 2492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:46:14.510977 2492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:46:14.512477 2492 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:46:14.512530 2492 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:14.512573 2492 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:46:14.512599 2492 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:14.522442 2492 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45131
I20260927 12:46:14.522482 2622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45131 every 8 connection(s)
Sep 27 12:46:14 dist-test-slave-lkn4 krb5kdc[2095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513174, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:46:14.523523 2492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:46:14.526708 2189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:14.513212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48415 (local address 127.0.98.62:41189)
0927 12:46:14.513343 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:14.513347 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:14.514438 (+ 1091us) server_negotiation.cc:407] Connection header received
0927 12:46:14.515476 (+ 1038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:14.515482 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:14.515543 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:14.515633 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:14.517371 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:14.518234 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:14.520000 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:14.520292 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:14.520394 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:14.523416 (+ 3022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:14.523420 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:14.523422 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:14.523448 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:14.524969 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:14.525433 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:14.525436 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:14.525437 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:14.525521 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:14.525775 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:14.525777 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:14.525778 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:14.525942 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:14.526011 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:14.526487 (+ 476us) server_negotiation.cc:299] Negotiation successful
0927 12:46:14.526590 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 12:46:14.527470 2516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:14.513792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41189 (local address 127.0.98.3:48415)
0927 12:46:14.514298 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:14.514339 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:14.515217 (+ 878us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:14.515962 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:14.515973 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:14.516476 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:14.517194 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:14.517214 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:14.518392 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:14.518396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:14.519855 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:14.519866 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:14.520459 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:14.520463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:14.520681 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:14.521398 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:14.521418 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:14.523239 (+ 1821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:14.525064 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:14.525069 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:14.525081 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:14.525324 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:14.525594 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:14.525596 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:14.525598 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:14.525711 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:14.526012 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:14.526016 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:14.526266 (+ 250us) client_negotiation.cc:769] Sending connection context
0927 12:46:14.526440 (+ 174us) client_negotiation.cc:240] Negotiation successful
0927 12:46:14.526684 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":165,"threads_started":1}
I20260927 12:46:14.528604 2623 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41189
I20260927 12:46:14.528761 2623 heartbeater.cc:464] Registering TS with master...
I20260927 12:46:14.529312 2623 heartbeater.cc:511] Master 127.0.98.62:41189 requested a full tablet report, sending...
I20260927 12:46:14.530427 2139 ts_manager.cc:194] Registered new tserver with Master: 2d94505a8b6a4daab8fdbdffdf354596 (127.0.98.3:45131)
I20260927 12:46:14.531116 2139 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48415
I20260927 12:46:14.531599 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 2492
I20260927 12:46:14.531698 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:46:14.531944 392 external_mini_cluster.cc:1428] Setting key 49b7faf15497435494f5be88f67ae96a
I20260927 12:46:14.533867 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 12:46:14.538358 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:14.530873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41189 (local address 127.0.98.3:59529)
0927 12:46:14.531123 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:14.531134 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:14.531226 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:14.531535 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:14.531537 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:14.531727 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:14.531871 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:14.531879 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:14.532777 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:14.532780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:14.533459 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:14.533468 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:14.534006 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:14.534010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:14.534126 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:14.534588 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:14.534602 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:14.535030 (+ 428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:14.536905 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:14.536908 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:14.536910 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:14.537111 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:14.537397 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:14.537400 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:14.537401 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:14.537482 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:14.537854 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:14.537856 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:14.537926 (+ 70us) client_negotiation.cc:769] Sending connection context
0927 12:46:14.538000 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 12:46:14.538171 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":4,"thread_start_us":89,"threads_started":1}
I20260927 12:46:14.538501 2189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:14.530928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59529 (local address 127.0.98.62:41189)
0927 12:46:14.531030 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:14.531035 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:14.531214 (+ 179us) server_negotiation.cc:407] Connection header received
0927 12:46:14.531299 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:14.531302 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:14.531344 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:14.531437 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:14.531986 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:14.532648 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:14.533561 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:14.533873 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:14.533968 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:14.535163 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:14.535167 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:14.535170 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:14.535192 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:14.536796 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:14.537203 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:14.537207 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:14.537209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:14.537293 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:14.537555 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:14.537558 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:14.537560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:14.537767 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:14.537862 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:14.538241 (+ 379us) server_negotiation.cc:299] Negotiation successful
0927 12:46:14.538356 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
Sep 27 12:46:14 dist-test-slave-lkn4 krb5kdc[2095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 12:46:14 dist-test-slave-lkn4 krb5kdc[2095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:46:14.558840 2189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:14.551290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55234 (local address 127.0.98.62:41189)
0927 12:46:14.551424 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:14.551427 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:14.551440 (+ 13us) server_negotiation.cc:407] Connection header received
0927 12:46:14.551478 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:14.551481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:14.551517 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:14.551582 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:14.552228 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:14.552812 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:14.553647 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:14.553930 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:14.554016 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:14.556215 (+ 2199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:14.556218 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:14.556220 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:14.556242 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:14.557512 (+ 1270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:14.557873 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:14.557875 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:14.557877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:14.557944 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:14.558190 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:14.558192 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:14.558193 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:14.558377 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:14.558476 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:14.558625 (+ 149us) server_negotiation.cc:299] Negotiation successful
0927 12:46:14.558716 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 12:46:14.561442 2139 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55234:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:46:14.564414 2139 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:46:14.578709 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:14.572855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45131 (local address 127.0.0.1:48880)
0927 12:46:14.573681 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:14.573735 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:14.573897 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:14.574684 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:14.574686 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:14.574700 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:46:14.574787 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:14.574793 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:14.575830 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:14.575833 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:14.577352 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:14.577359 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:14.578161 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:14.578164 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:14.578288 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:14.578329 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 12:46:14.578434 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 12:46:14.578561 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"mutex_wait_us":86,"thread_start_us":60,"threads_started":1}
I20260927 12:46:14.578722 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:14.572946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48880 (local address 127.0.98.3:45131)
0927 12:46:14.573754 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:14.573767 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:14.574033 (+ 266us) server_negotiation.cc:407] Connection header received
0927 12:46:14.574147 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:14.574153 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:14.574372 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:14.574524 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:46:14.574898 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:14.575722 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:14.577454 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:14.578027 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:14.578154 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:14.578333 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:14.578471 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 12:46:14.578565 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":707,"thread_start_us":97,"threads_started":1}
I20260927 12:46:14.579123 2640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:14.573008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45249 (local address 127.0.0.1:36378)
0927 12:46:14.573848 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:14.573862 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:14.573948 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:14.574512 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:14.574518 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:14.574553 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:46:14.574688 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:14.574694 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:14.575963 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:14.575966 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:14.577333 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:14.577343 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:14.578682 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:14.578685 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:14.578816 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:14.578841 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 12:46:14.578913 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 12:46:14.578981 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":775,"mutex_wait_us":38,"thread_start_us":96,"threads_started":1}
I20260927 12:46:14.579195 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:14.573613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36378 (local address 127.0.98.2:45249)
0927 12:46:14.574117 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:14.574128 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:14.574147 (+ 19us) server_negotiation.cc:407] Connection header received
0927 12:46:14.574206 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:14.574210 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:14.574372 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:14.574524 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:46:14.574805 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:14.575860 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:14.577513 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:14.578372 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:14.578471 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:14.578622 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:14.578964 (+ 342us) server_negotiation.cc:299] Negotiation successful
0927 12:46:14.579060 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":74,"threads_started":1}
I20260927 12:46:14.579252 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:14.573109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39256 (local address 127.0.98.1:37687)
0927 12:46:14.573394 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:14.573407 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:14.574246 (+ 839us) server_negotiation.cc:407] Connection header received
0927 12:46:14.574451 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:14.574457 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:14.574663 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:14.574788 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:46:14.575181 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:14.576156 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:14.577512 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:14.578372 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:14.578471 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:14.578622 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:14.579028 (+ 406us) server_negotiation.cc:299] Negotiation successful
0927 12:46:14.579120 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":79,"threads_started":1}
I20260927 12:46:14.580376 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:14.572855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37687 (local address 127.0.0.1:39256)
0927 12:46:14.574149 (+ 1294us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:14.574160 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:14.574311 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:14.574961 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:14.574964 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:14.574980 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:46:14.575064 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:14.575070 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:14.576257 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:14.576260 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:14.577357 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:14.577366 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:14.578681 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:14.578685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:14.578893 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:14.578914 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 12:46:14.580173 (+ 1259us) client_negotiation.cc:240] Negotiation successful
0927 12:46:14.580251 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1242,"mutex_wait_us":865,"thread_start_us":88,"threads_started":1}
I20260927 12:46:14.582340 2417 tablet_service.cc:1467] Processing CreateTablet for tablet d897625dc49c462fb17b027944540640 (DEFAULT_TABLE table=test-table [id=65a1e66cd92842d29f2c0496ad5e1b07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:46:14.583003 2277 tablet_service.cc:1467] Processing CreateTablet for tablet d897625dc49c462fb17b027944540640 (DEFAULT_TABLE table=test-table [id=65a1e66cd92842d29f2c0496ad5e1b07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:46:14.583480 2417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d897625dc49c462fb17b027944540640. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:46:14.583591 2557 tablet_service.cc:1467] Processing CreateTablet for tablet d897625dc49c462fb17b027944540640 (DEFAULT_TABLE table=test-table [id=65a1e66cd92842d29f2c0496ad5e1b07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:46:14.584030 2277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d897625dc49c462fb17b027944540640. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:46:14.584393 2557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d897625dc49c462fb17b027944540640. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:46:14.590461 2646 tablet_bootstrap.cc:492] T d897625dc49c462fb17b027944540640 P 2d94505a8b6a4daab8fdbdffdf354596: Bootstrap starting.
I20260927 12:46:14.592046 2648 tablet_bootstrap.cc:492] T d897625dc49c462fb17b027944540640 P c60eee98017b49d4aa748b9ac4846cde: Bootstrap starting.
I20260927 12:46:14.592178 2647 tablet_bootstrap.cc:492] T d897625dc49c462fb17b027944540640 P 1f0060ce040449f1a4415d377932fa44: Bootstrap starting.
I20260927 12:46:14.593339 2646 tablet_bootstrap.cc:654] T d897625dc49c462fb17b027944540640 P 2d94505a8b6a4daab8fdbdffdf354596: Neither blocks nor log segments found. Creating new log.
I20260927 12:46:14.593989 2647 tablet_bootstrap.cc:654] T d897625dc49c462fb17b027944540640 P 1f0060ce040449f1a4415d377932fa44: Neither blocks nor log segments found. Creating new log.
I20260927 12:46:14.594164 2648 tablet_bootstrap.cc:654] T d897625dc49c462fb17b027944540640 P c60eee98017b49d4aa748b9ac4846cde: Neither blocks nor log segments found. Creating new log.
I20260927 12:46:14.594239 2646 log.cc:824] T d897625dc49c462fb17b027944540640 P 2d94505a8b6a4daab8fdbdffdf354596: Log is configured to *not* fsync() on all Append() calls
I20260927 12:46:14.594758 2647 log.cc:824] T d897625dc49c462fb17b027944540640 P 1f0060ce040449f1a4415d377932fa44: Log is configured to *not* fsync() on all Append() calls
I20260927 12:46:14.594895 2648 log.cc:824] T d897625dc49c462fb17b027944540640 P c60eee98017b49d4aa748b9ac4846cde: Log is configured to *not* fsync() on all Append() calls
I20260927 12:46:14.596370 2646 tablet_bootstrap.cc:492] T d897625dc49c462fb17b027944540640 P 2d94505a8b6a4daab8fdbdffdf354596: No bootstrap required, opened a new log
I20260927 12:46:14.596470 2648 tablet_bootstrap.cc:492] T d897625dc49c462fb17b027944540640 P c60eee98017b49d4aa748b9ac4846cde: No bootstrap required, opened a new log
I20260927 12:46:14.596566 2647 tablet_bootstrap.cc:492] T d897625dc49c462fb17b027944540640 P 1f0060ce040449f1a4415d377932fa44: No bootstrap required, opened a new log
I20260927 12:46:14.596570 2646 ts_tablet_manager.cc:1397] T d897625dc49c462fb17b027944540640 P 2d94505a8b6a4daab8fdbdffdf354596: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 12:46:14.596650 2648 ts_tablet_manager.cc:1397] T d897625dc49c462fb17b027944540640 P c60eee98017b49d4aa748b9ac4846cde: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260927 12:46:14.596719 2647 ts_tablet_manager.cc:1397] T d897625dc49c462fb17b027944540640 P 1f0060ce040449f1a4415d377932fa44: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 12:46:14.599881 2648 raft_consensus.cc:348] T d897625dc49c462fb17b027944540640 P c60eee98017b49d4aa748b9ac4846cde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d94505a8b6a4daab8fdbdffdf354596" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45131 } } peers { permanent_uuid: "c60eee98017b49d4aa748b9ac4846cde" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37687 } } peers { permanent_uuid: "1f0060ce040449f1a4415d377932fa44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45249 } }
I20260927 12:46:14.600097 2648 raft_consensus.cc:374] T d897625dc49c462fb17b027944540640 P c60eee98017b49d4aa748b9ac4846cde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:46:14.600126 2648 raft_consensus.cc:729] T d897625dc49c462fb17b027944540640 P c60eee98017b49d4aa748b9ac4846cde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c60eee98017b49d4aa748b9ac4846cde, State: Initialized, Role: FOLLOWER
I20260927 12:46:14.600497 2648 consensus_queue.cc:260] T d897625dc49c462fb17b027944540640 P c60eee98017b49d4aa748b9ac4846cde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d94505a8b6a4daab8fdbdffdf354596" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45131 } } peers { permanent_uuid: "c60eee98017b49d4aa748b9ac4846cde" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37687 } } peers { permanent_uuid: "1f0060ce040449f1a4415d377932fa44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45249 } }
I20260927 12:46:14.600432 2646 raft_consensus.cc:348] T d897625dc49c462fb17b027944540640 P 2d94505a8b6a4daab8fdbdffdf354596 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d94505a8b6a4daab8fdbdffdf354596" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45131 } } peers { permanent_uuid: "c60eee98017b49d4aa748b9ac4846cde" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37687 } } peers { permanent_uuid: "1f0060ce040449f1a4415d377932fa44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45249 } }
I20260927 12:46:14.600432 2647 raft_consensus.cc:348] T d897625dc49c462fb17b027944540640 P 1f0060ce040449f1a4415d377932fa44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d94505a8b6a4daab8fdbdffdf354596" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45131 } } peers { permanent_uuid: "c60eee98017b49d4aa748b9ac4846cde" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37687 } } peers { permanent_uuid: "1f0060ce040449f1a4415d377932fa44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45249 } }
I20260927 12:46:14.600672 2647 raft_consensus.cc:374] T d897625dc49c462fb17b027944540640 P 1f0060ce040449f1a4415d377932fa44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:46:14.600672 2646 raft_consensus.cc:374] T d897625dc49c462fb17b027944540640 P 2d94505a8b6a4daab8fdbdffdf354596 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:46:14.600716 2646 raft_consensus.cc:729] T d897625dc49c462fb17b027944540640 P 2d94505a8b6a4daab8fdbdffdf354596 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d94505a8b6a4daab8fdbdffdf354596, State: Initialized, Role: FOLLOWER
I20260927 12:46:14.600726 2647 raft_consensus.cc:729] T d897625dc49c462fb17b027944540640 P 1f0060ce040449f1a4415d377932fa44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0060ce040449f1a4415d377932fa44, State: Initialized, Role: FOLLOWER
I20260927 12:46:14.601114 2343 heartbeater.cc:503] Master 127.0.98.62:41189 was elected leader, sending a full tablet report...
I20260927 12:46:14.601480 2648 ts_tablet_manager.cc:1428] T d897625dc49c462fb17b027944540640 P c60eee98017b49d4aa748b9ac4846cde: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260927 12:46:14.601414 2647 consensus_queue.cc:260] T d897625dc49c462fb17b027944540640 P 1f0060ce040449f1a4415d377932fa44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d94505a8b6a4daab8fdbdffdf354596" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45131 } } peers { permanent_uuid: "c60eee98017b49d4aa748b9ac4846cde" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37687 } } peers { permanent_uuid: "1f0060ce040449f1a4415d377932fa44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45249 } }
I20260927 12:46:14.602017 2483 heartbeater.cc:503] Master 127.0.98.62:41189 was elected leader, sending a full tablet report...
I20260927 12:46:14.602175 2653 raft_consensus.cc:482] T d897625dc49c462fb17b027944540640 P 1f0060ce040449f1a4415d377932fa44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:46:14.602246 2646 consensus_queue.cc:260] T d897625dc49c462fb17b027944540640 P 2d94505a8b6a4daab8fdbdffdf354596 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d94505a8b6a4daab8fdbdffdf354596" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45131 } } peers { permanent_uuid: "c60eee98017b49d4aa748b9ac4846cde" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37687 } } peers { permanent_uuid: "1f0060ce040449f1a4415d377932fa44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45249 } }
I20260927 12:46:14.602334 2653 raft_consensus.cc:504] T d897625dc49c462fb17b027944540640 P 1f0060ce040449f1a4415d377932fa44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d94505a8b6a4daab8fdbdffdf354596" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45131 } } peers { permanent_uuid: "c60eee98017b49d4aa748b9ac4846cde" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37687 } } peers { permanent_uuid: "1f0060ce040449f1a4415d377932fa44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45249 } }
I20260927 12:46:14.602408 2647 ts_tablet_manager.cc:1428] T d897625dc49c462fb17b027944540640 P 1f0060ce040449f1a4415d377932fa44: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 12:46:14.602944 2623 heartbeater.cc:503] Master 127.0.98.62:41189 was elected leader, sending a full tablet report...
I20260927 12:46:14.603154 2646 ts_tablet_manager.cc:1428] T d897625dc49c462fb17b027944540640 P 2d94505a8b6a4daab8fdbdffdf354596: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260927 12:46:14.603811 2653 leader_election.cc:288] T d897625dc49c462fb17b027944540640 P 1f0060ce040449f1a4415d377932fa44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d94505a8b6a4daab8fdbdffdf354596 (127.0.98.3:45131), c60eee98017b49d4aa748b9ac4846cde (127.0.98.1:37687)
I20260927 12:46:14.608779 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:14.604369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38843 (local address 127.0.98.3:45131)
0927 12:46:14.604469 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:14.604473 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:14.605034 (+ 561us) server_negotiation.cc:407] Connection header received
0927 12:46:14.605084 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:14.605087 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:14.605134 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:14.605197 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:46:14.605559 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:14.606281 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:14.607650 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:14.608238 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:14.608342 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:14.608406 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:14.608587 (+ 181us) server_negotiation.cc:299] Negotiation successful
0927 12:46:14.608659 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 12:46:14.608780 2376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:14.603594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45131 (local address 127.0.98.2:38843)
0927 12:46:14.604448 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:14.604465 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:14.604558 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:14.605307 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:14.605311 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:14.605330 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:46:14.605438 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:14.605445 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:14.606399 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:14.606402 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:14.607541 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:14.607550 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:14.608344 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:14.608347 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:14.608435 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:14.608496 (+ 61us) client_negotiation.cc:769] Sending connection context
0927 12:46:14.608567 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 12:46:14.608646 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":767,"mutex_wait_us":45}
I20260927 12:46:14.609381 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d897625dc49c462fb17b027944540640" candidate_uuid: "1f0060ce040449f1a4415d377932fa44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d94505a8b6a4daab8fdbdffdf354596" is_pre_election: true
I20260927 12:46:14.609642 2577 raft_consensus.cc:2454] T d897625dc49c462fb17b027944540640 P 2d94505a8b6a4daab8fdbdffdf354596 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f0060ce040449f1a4415d377932fa44 in term 0.
I20260927 12:46:14.610116 2368 leader_election.cc:302] T d897625dc49c462fb17b027944540640 P 1f0060ce040449f1a4415d377932fa44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0060ce040449f1a4415d377932fa44, 2d94505a8b6a4daab8fdbdffdf354596; no voters:
I20260927 12:46:14.610503 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:14.604899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39111 (local address 127.0.98.1:37687)
0927 12:46:14.605000 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:14.605004 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:14.606525 (+ 1521us) server_negotiation.cc:407] Connection header received
0927 12:46:14.606601 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:14.606604 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:14.606648 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:14.606718 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:46:14.607488 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:14.608185 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:14.609436 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:14.609947 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:14.610068 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:14.610126 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:14.610333 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 12:46:14.610405 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 12:46:14.610522 2655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:14.604788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37687 (local address 127.0.98.2:39111)
0927 12:46:14.606431 (+ 1643us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:14.606444 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:14.606529 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:14.606829 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:14.606831 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:14.606848 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:46:14.607361 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:14.607371 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:14.608319 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:14.608322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:14.609335 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:14.609343 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:14.610100 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:14.610105 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:14.610194 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:14.610214 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 12:46:14.610314 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 12:46:14.610381 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1572,"mutex_wait_us":2,"thread_start_us":113,"threads_started":1}
I20260927 12:46:14.610656 2653 raft_consensus.cc:2781] T d897625dc49c462fb17b027944540640 P 1f0060ce040449f1a4415d377932fa44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:46:14.610725 2653 raft_consensus.cc:482] T d897625dc49c462fb17b027944540640 P 1f0060ce040449f1a4415d377932fa44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:46:14.610766 2653 raft_consensus.cc:3037] T d897625dc49c462fb17b027944540640 P 1f0060ce040449f1a4415d377932fa44 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:46:14.611054 2297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d897625dc49c462fb17b027944540640" candidate_uuid: "1f0060ce040449f1a4415d377932fa44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c60eee98017b49d4aa748b9ac4846cde" is_pre_election: true
I20260927 12:46:14.611287 2297 raft_consensus.cc:2454] T d897625dc49c462fb17b027944540640 P c60eee98017b49d4aa748b9ac4846cde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f0060ce040449f1a4415d377932fa44 in term 0.
I20260927 12:46:14.611846 2653 raft_consensus.cc:504] T d897625dc49c462fb17b027944540640 P 1f0060ce040449f1a4415d377932fa44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d94505a8b6a4daab8fdbdffdf354596" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45131 } } peers { permanent_uuid: "c60eee98017b49d4aa748b9ac4846cde" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37687 } } peers { permanent_uuid: "1f0060ce040449f1a4415d377932fa44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45249 } }
I20260927 12:46:14.612149 2653 leader_election.cc:288] T d897625dc49c462fb17b027944540640 P 1f0060ce040449f1a4415d377932fa44 [CANDIDATE]: Term 1 election: Requested vote from peers 2d94505a8b6a4daab8fdbdffdf354596 (127.0.98.3:45131), c60eee98017b49d4aa748b9ac4846cde (127.0.98.1:37687)
I20260927 12:46:14.612474 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d897625dc49c462fb17b027944540640" candidate_uuid: "1f0060ce040449f1a4415d377932fa44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d94505a8b6a4daab8fdbdffdf354596"
I20260927 12:46:14.612493 2297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d897625dc49c462fb17b027944540640" candidate_uuid: "1f0060ce040449f1a4415d377932fa44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c60eee98017b49d4aa748b9ac4846cde"
I20260927 12:46:14.612591 2577 raft_consensus.cc:3037] T d897625dc49c462fb17b027944540640 P 2d94505a8b6a4daab8fdbdffdf354596 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:46:14.612592 2297 raft_consensus.cc:3037] T d897625dc49c462fb17b027944540640 P c60eee98017b49d4aa748b9ac4846cde [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:46:14.613791 2577 raft_consensus.cc:2454] T d897625dc49c462fb17b027944540640 P 2d94505a8b6a4daab8fdbdffdf354596 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f0060ce040449f1a4415d377932fa44 in term 1.
I20260927 12:46:14.613791 2297 raft_consensus.cc:2454] T d897625dc49c462fb17b027944540640 P c60eee98017b49d4aa748b9ac4846cde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f0060ce040449f1a4415d377932fa44 in term 1.
I20260927 12:46:14.614096 2370 leader_election.cc:302] T d897625dc49c462fb17b027944540640 P 1f0060ce040449f1a4415d377932fa44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0060ce040449f1a4415d377932fa44, c60eee98017b49d4aa748b9ac4846cde; no voters:
I20260927 12:46:14.614264 2653 raft_consensus.cc:2781] T d897625dc49c462fb17b027944540640 P 1f0060ce040449f1a4415d377932fa44 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:46:14.614460 2653 raft_consensus.cc:686] T d897625dc49c462fb17b027944540640 P 1f0060ce040449f1a4415d377932fa44 [term 1 LEADER]: Becoming Leader. State: Replica: 1f0060ce040449f1a4415d377932fa44, State: Running, Role: LEADER
I20260927 12:46:14.614758 2653 consensus_queue.cc:237] T d897625dc49c462fb17b027944540640 P 1f0060ce040449f1a4415d377932fa44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d94505a8b6a4daab8fdbdffdf354596" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45131 } } peers { permanent_uuid: "c60eee98017b49d4aa748b9ac4846cde" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37687 } } peers { permanent_uuid: "1f0060ce040449f1a4415d377932fa44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45249 } }
I20260927 12:46:14.617661 2137 catalog_manager.cc:5495] T d897625dc49c462fb17b027944540640 P 1f0060ce040449f1a4415d377932fa44 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f0060ce040449f1a4415d377932fa44 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "1f0060ce040449f1a4415d377932fa44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d94505a8b6a4daab8fdbdffdf354596" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c60eee98017b49d4aa748b9ac4846cde" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f0060ce040449f1a4415d377932fa44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45249 } health_report { overall_health: HEALTHY } } }
Sep 27 12:46:14 dist-test-slave-lkn4 krb5kdc[2095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513173, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:46:14.655735 2189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:14.647782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55242 (local address 127.0.98.62:41189)
0927 12:46:14.647906 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:14.647910 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:14.648117 (+ 207us) server_negotiation.cc:407] Connection header received
0927 12:46:14.648244 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:14.648247 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:14.648288 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:14.648346 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:14.648956 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:14.649519 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:14.650289 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:14.650520 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:14.650599 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:14.652792 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:14.652796 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:14.652798 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:14.652818 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:14.654175 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:14.654607 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:14.654610 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:14.654612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:14.654681 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:14.654903 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:14.654906 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:14.654908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:14.655073 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:14.655152 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:14.655432 (+ 280us) server_negotiation.cc:299] Negotiation successful
0927 12:46:14.655591 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":41}
I20260927 12:46:14.655735 2663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:14.647737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41189 (local address 127.0.0.1:55242)
0927 12:46:14.648037 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:14.648053 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:14.648155 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:14.648370 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:14.648372 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:14.648759 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:14.648858 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:14.648865 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:14.649604 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:14.649607 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:14.650191 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:14.650199 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:14.650613 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:14.650615 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:14.650678 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:14.651236 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:14.651247 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:14.652605 (+ 1358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:14.654270 (+ 1665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:14.654281 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:14.654288 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:14.654529 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:14.654752 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:14.654754 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:14.654756 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:14.654833 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:14.655160 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:14.655168 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:14.655284 (+ 116us) client_negotiation.cc:769] Sending connection context
0927 12:46:14.655379 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 12:46:14.655539 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":128,"threads_started":1}
I20260927 12:46:14.659710 2137 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55242:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 12:46:14.660444 2137 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:46:14.665045 2417 tablet_service.cc:1467] Processing CreateTablet for tablet 561fae25418b4177be9143dc766e6387 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a21122ce042f4572af86dfb8621a95a0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 12:46:14.665027 2277 tablet_service.cc:1467] Processing CreateTablet for tablet 561fae25418b4177be9143dc766e6387 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a21122ce042f4572af86dfb8621a95a0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 12:46:14.665263 2557 tablet_service.cc:1467] Processing CreateTablet for tablet 561fae25418b4177be9143dc766e6387 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a21122ce042f4572af86dfb8621a95a0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 12:46:14.665812 2557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 561fae25418b4177be9143dc766e6387. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:46:14.665819 2417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 561fae25418b4177be9143dc766e6387. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:46:14.665820 2277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 561fae25418b4177be9143dc766e6387. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:46:14.668511 2648 tablet_bootstrap.cc:492] T 561fae25418b4177be9143dc766e6387 P c60eee98017b49d4aa748b9ac4846cde: Bootstrap starting.
I20260927 12:46:14.668607 2646 tablet_bootstrap.cc:492] T 561fae25418b4177be9143dc766e6387 P 2d94505a8b6a4daab8fdbdffdf354596: Bootstrap starting.
I20260927 12:46:14.669381 2648 tablet_bootstrap.cc:654] T 561fae25418b4177be9143dc766e6387 P c60eee98017b49d4aa748b9ac4846cde: Neither blocks nor log segments found. Creating new log.
I20260927 12:46:14.669443 2646 tablet_bootstrap.cc:654] T 561fae25418b4177be9143dc766e6387 P 2d94505a8b6a4daab8fdbdffdf354596: Neither blocks nor log segments found. Creating new log.
I20260927 12:46:14.669983 2647 tablet_bootstrap.cc:492] T 561fae25418b4177be9143dc766e6387 P 1f0060ce040449f1a4415d377932fa44: Bootstrap starting.
I20260927 12:46:14.670549 2648 tablet_bootstrap.cc:492] T 561fae25418b4177be9143dc766e6387 P c60eee98017b49d4aa748b9ac4846cde: No bootstrap required, opened a new log
I20260927 12:46:14.670624 2648 ts_tablet_manager.cc:1397] T 561fae25418b4177be9143dc766e6387 P c60eee98017b49d4aa748b9ac4846cde: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260927 12:46:14.670866 2647 tablet_bootstrap.cc:654] T 561fae25418b4177be9143dc766e6387 P 1f0060ce040449f1a4415d377932fa44: Neither blocks nor log segments found. Creating new log.
I20260927 12:46:14.671032 2648 raft_consensus.cc:348] T 561fae25418b4177be9143dc766e6387 P c60eee98017b49d4aa748b9ac4846cde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60eee98017b49d4aa748b9ac4846cde" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37687 } } peers { permanent_uuid: "1f0060ce040449f1a4415d377932fa44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45249 } } peers { permanent_uuid: "2d94505a8b6a4daab8fdbdffdf354596" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45131 } }
I20260927 12:46:14.671134 2648 raft_consensus.cc:374] T 561fae25418b4177be9143dc766e6387 P c60eee98017b49d4aa748b9ac4846cde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:46:14.671156 2648 raft_consensus.cc:729] T 561fae25418b4177be9143dc766e6387 P c60eee98017b49d4aa748b9ac4846cde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c60eee98017b49d4aa748b9ac4846cde, State: Initialized, Role: FOLLOWER
I20260927 12:46:14.671305 2648 consensus_queue.cc:260] T 561fae25418b4177be9143dc766e6387 P c60eee98017b49d4aa748b9ac4846cde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60eee98017b49d4aa748b9ac4846cde" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37687 } } peers { permanent_uuid: "1f0060ce040449f1a4415d377932fa44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45249 } } peers { permanent_uuid: "2d94505a8b6a4daab8fdbdffdf354596" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45131 } }
I20260927 12:46:14.671567 2648 ts_tablet_manager.cc:1428] T 561fae25418b4177be9143dc766e6387 P c60eee98017b49d4aa748b9ac4846cde: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260927 12:46:14.671772 2646 tablet_bootstrap.cc:492] T 561fae25418b4177be9143dc766e6387 P 2d94505a8b6a4daab8fdbdffdf354596: No bootstrap required, opened a new log
I20260927 12:46:14.671865 2646 ts_tablet_manager.cc:1397] T 561fae25418b4177be9143dc766e6387 P 2d94505a8b6a4daab8fdbdffdf354596: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 12:46:14.671813 2652 tablet_replica.cc:439] T 561fae25418b4177be9143dc766e6387 P c60eee98017b49d4aa748b9ac4846cde: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60eee98017b49d4aa748b9ac4846cde" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37687 } } peers { permanent_uuid: "1f0060ce040449f1a4415d377932fa44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45249 } } peers { permanent_uuid: "2d94505a8b6a4daab8fdbdffdf354596" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45131 } } }
I20260927 12:46:14.671943 2652 tablet_replica.cc:442] T 561fae25418b4177be9143dc766e6387 P c60eee98017b49d4aa748b9ac4846cde: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 12:46:14.672245 2646 raft_consensus.cc:348] T 561fae25418b4177be9143dc766e6387 P 2d94505a8b6a4daab8fdbdffdf354596 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60eee98017b49d4aa748b9ac4846cde" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37687 } } peers { permanent_uuid: "1f0060ce040449f1a4415d377932fa44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45249 } } peers { permanent_uuid: "2d94505a8b6a4daab8fdbdffdf354596" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45131 } }
I20260927 12:46:14.672338 2646 raft_consensus.cc:374] T 561fae25418b4177be9143dc766e6387 P 2d94505a8b6a4daab8fdbdffdf354596 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:46:14.672389 2646 raft_consensus.cc:729] T 561fae25418b4177be9143dc766e6387 P 2d94505a8b6a4daab8fdbdffdf354596 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d94505a8b6a4daab8fdbdffdf354596, State: Initialized, Role: FOLLOWER
I20260927 12:46:14.672508 2646 consensus_queue.cc:260] T 561fae25418b4177be9143dc766e6387 P 2d94505a8b6a4daab8fdbdffdf354596 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60eee98017b49d4aa748b9ac4846cde" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37687 } } peers { permanent_uuid: "1f0060ce040449f1a4415d377932fa44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45249 } } peers { permanent_uuid: "2d94505a8b6a4daab8fdbdffdf354596" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45131 } }
I20260927 12:46:14.672758 2646 ts_tablet_manager.cc:1428] T 561fae25418b4177be9143dc766e6387 P 2d94505a8b6a4daab8fdbdffdf354596: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 12:46:14.672983 2654 tablet_replica.cc:439] T 561fae25418b4177be9143dc766e6387 P 2d94505a8b6a4daab8fdbdffdf354596: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60eee98017b49d4aa748b9ac4846cde" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37687 } } peers { permanent_uuid: "1f0060ce040449f1a4415d377932fa44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45249 } } peers { permanent_uuid: "2d94505a8b6a4daab8fdbdffdf354596" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45131 } } }
I20260927 12:46:14.673112 2654 tablet_replica.cc:442] T 561fae25418b4177be9143dc766e6387 P 2d94505a8b6a4daab8fdbdffdf354596: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 12:46:14.673622 2647 tablet_bootstrap.cc:492] T 561fae25418b4177be9143dc766e6387 P 1f0060ce040449f1a4415d377932fa44: No bootstrap required, opened a new log
I20260927 12:46:14.673715 2647 ts_tablet_manager.cc:1397] T 561fae25418b4177be9143dc766e6387 P 1f0060ce040449f1a4415d377932fa44: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260927 12:46:14.674187 2647 raft_consensus.cc:348] T 561fae25418b4177be9143dc766e6387 P 1f0060ce040449f1a4415d377932fa44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60eee98017b49d4aa748b9ac4846cde" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37687 } } peers { permanent_uuid: "1f0060ce040449f1a4415d377932fa44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45249 } } peers { permanent_uuid: "2d94505a8b6a4daab8fdbdffdf354596" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45131 } }
I20260927 12:46:14.674300 2647 raft_consensus.cc:374] T 561fae25418b4177be9143dc766e6387 P 1f0060ce040449f1a4415d377932fa44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:46:14.674335 2647 raft_consensus.cc:729] T 561fae25418b4177be9143dc766e6387 P 1f0060ce040449f1a4415d377932fa44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0060ce040449f1a4415d377932fa44, State: Initialized, Role: FOLLOWER
I20260927 12:46:14.674439 2647 consensus_queue.cc:260] T 561fae25418b4177be9143dc766e6387 P 1f0060ce040449f1a4415d377932fa44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60eee98017b49d4aa748b9ac4846cde" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37687 } } peers { permanent_uuid: "1f0060ce040449f1a4415d377932fa44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45249 } } peers { permanent_uuid: "2d94505a8b6a4daab8fdbdffdf354596" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45131 } }
I20260927 12:46:14.674695 2647 ts_tablet_manager.cc:1428] T 561fae25418b4177be9143dc766e6387 P 1f0060ce040449f1a4415d377932fa44: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 12:46:14.674685 2653 tablet_replica.cc:439] T 561fae25418b4177be9143dc766e6387 P 1f0060ce040449f1a4415d377932fa44: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60eee98017b49d4aa748b9ac4846cde" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37687 } } peers { permanent_uuid: "1f0060ce040449f1a4415d377932fa44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45249 } } peers { permanent_uuid: "2d94505a8b6a4daab8fdbdffdf354596" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45131 } } }
I20260927 12:46:14.674809 2653 tablet_replica.cc:442] T 561fae25418b4177be9143dc766e6387 P 1f0060ce040449f1a4415d377932fa44: This TxnStatusTablet replica's current role is: FOLLOWER
W20260927 12:46:14.774799 2624 tablet.cc:2369] T 561fae25418b4177be9143dc766e6387 P 2d94505a8b6a4daab8fdbdffdf354596: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 12:46:14.816659 2484 tablet.cc:2369] T 561fae25418b4177be9143dc766e6387 P 1f0060ce040449f1a4415d377932fa44: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 12:46:14.837793 2344 tablet.cc:2369] T 561fae25418b4177be9143dc766e6387 P c60eee98017b49d4aa748b9ac4846cde: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:46:14.885416 2654 raft_consensus.cc:482] T 561fae25418b4177be9143dc766e6387 P 2d94505a8b6a4daab8fdbdffdf354596 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:46:14.885561 2654 raft_consensus.cc:504] T 561fae25418b4177be9143dc766e6387 P 2d94505a8b6a4daab8fdbdffdf354596 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60eee98017b49d4aa748b9ac4846cde" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37687 } } peers { permanent_uuid: "1f0060ce040449f1a4415d377932fa44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45249 } } peers { permanent_uuid: "2d94505a8b6a4daab8fdbdffdf354596" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45131 } }
I20260927 12:46:14.886684 2654 leader_election.cc:288] T 561fae25418b4177be9143dc766e6387 P 2d94505a8b6a4daab8fdbdffdf354596 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c60eee98017b49d4aa748b9ac4846cde (127.0.98.1:37687), 1f0060ce040449f1a4415d377932fa44 (127.0.98.2:45249)
I20260927 12:46:14.890702 2516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:14.886925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37687 (local address 127.0.98.3:54027)
0927 12:46:14.887028 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:14.887043 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:14.887135 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:14.887376 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:14.887379 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:14.887399 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:46:14.887499 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:14.887505 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:14.888434 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:14.888436 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:14.889586 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:14.889596 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:14.890368 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:14.890371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:14.890442 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:14.890461 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 12:46:14.890517 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 12:46:14.890572 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 12:46:14.890741 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:14.887013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54027 (local address 127.0.98.1:37687)
0927 12:46:14.887135 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:14.887139 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:14.887155 (+ 16us) server_negotiation.cc:407] Connection header received
0927 12:46:14.887213 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:14.887216 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:14.887280 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:14.887352 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:46:14.887642 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:14.888326 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:14.889703 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:14.890236 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:14.890360 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:14.890424 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:14.890546 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 12:46:14.890617 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 12:46:14.891075 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:14.887049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48625 (local address 127.0.98.2:45249)
0927 12:46:14.887175 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:14.887180 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:14.887575 (+ 395us) server_negotiation.cc:407] Connection header received
0927 12:46:14.887654 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:14.887657 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:14.887705 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:14.887785 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:46:14.888047 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:14.888625 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:14.889935 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:14.890537 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:14.890630 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:14.890682 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:14.890901 (+ 219us) server_negotiation.cc:299] Negotiation successful
0927 12:46:14.890975 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 12:46:14.891161 2652 raft_consensus.cc:482] T 561fae25418b4177be9143dc766e6387 P c60eee98017b49d4aa748b9ac4846cde [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:46:14.891271 2652 raft_consensus.cc:504] T 561fae25418b4177be9143dc766e6387 P c60eee98017b49d4aa748b9ac4846cde [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60eee98017b49d4aa748b9ac4846cde" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37687 } } peers { permanent_uuid: "1f0060ce040449f1a4415d377932fa44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45249 } } peers { permanent_uuid: "2d94505a8b6a4daab8fdbdffdf354596" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45131 } }
I20260927 12:46:14.891322 2297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "561fae25418b4177be9143dc766e6387" candidate_uuid: "2d94505a8b6a4daab8fdbdffdf354596" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c60eee98017b49d4aa748b9ac4846cde" is_pre_election: true
I20260927 12:46:14.891454 2667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:14.886942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45249 (local address 127.0.98.3:48625)
0927 12:46:14.887493 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:14.887503 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:14.887581 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:14.887829 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:14.887831 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:14.887849 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:46:14.887953 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:14.887957 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:14.888738 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:14.888742 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:14.889839 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:14.889846 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:14.890654 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:14.890658 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:14.890737 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:14.890758 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 12:46:14.891259 (+ 501us) client_negotiation.cc:240] Negotiation successful
0927 12:46:14.891331 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"mutex_wait_us":251,"thread_start_us":120,"threads_started":1}
I20260927 12:46:14.891697 2297 raft_consensus.cc:2454] T 561fae25418b4177be9143dc766e6387 P c60eee98017b49d4aa748b9ac4846cde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d94505a8b6a4daab8fdbdffdf354596 in term 0.
I20260927 12:46:14.892015 2437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "561fae25418b4177be9143dc766e6387" candidate_uuid: "2d94505a8b6a4daab8fdbdffdf354596" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0060ce040449f1a4415d377932fa44" is_pre_election: true
I20260927 12:46:14.892076 2510 leader_election.cc:302] T 561fae25418b4177be9143dc766e6387 P 2d94505a8b6a4daab8fdbdffdf354596 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d94505a8b6a4daab8fdbdffdf354596, c60eee98017b49d4aa748b9ac4846cde; no voters:
I20260927 12:46:14.892218 2437 raft_consensus.cc:2454] T 561fae25418b4177be9143dc766e6387 P 1f0060ce040449f1a4415d377932fa44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d94505a8b6a4daab8fdbdffdf354596 in term 0.
I20260927 12:46:14.892398 2654 raft_consensus.cc:2781] T 561fae25418b4177be9143dc766e6387 P 2d94505a8b6a4daab8fdbdffdf354596 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:46:14.892458 2654 raft_consensus.cc:482] T 561fae25418b4177be9143dc766e6387 P 2d94505a8b6a4daab8fdbdffdf354596 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:46:14.892493 2654 raft_consensus.cc:3037] T 561fae25418b4177be9143dc766e6387 P 2d94505a8b6a4daab8fdbdffdf354596 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:46:14.892581 2652 leader_election.cc:288] T 561fae25418b4177be9143dc766e6387 P c60eee98017b49d4aa748b9ac4846cde [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f0060ce040449f1a4415d377932fa44 (127.0.98.2:45249), 2d94505a8b6a4daab8fdbdffdf354596 (127.0.98.3:45131)
I20260927 12:46:14.893785 2654 raft_consensus.cc:504] T 561fae25418b4177be9143dc766e6387 P 2d94505a8b6a4daab8fdbdffdf354596 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60eee98017b49d4aa748b9ac4846cde" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37687 } } peers { permanent_uuid: "1f0060ce040449f1a4415d377932fa44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45249 } } peers { permanent_uuid: "2d94505a8b6a4daab8fdbdffdf354596" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45131 } }
I20260927 12:46:14.894181 2654 leader_election.cc:288] T 561fae25418b4177be9143dc766e6387 P 2d94505a8b6a4daab8fdbdffdf354596 [CANDIDATE]: Term 1 election: Requested vote from peers c60eee98017b49d4aa748b9ac4846cde (127.0.98.1:37687), 1f0060ce040449f1a4415d377932fa44 (127.0.98.2:45249)
I20260927 12:46:14.894619 2297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "561fae25418b4177be9143dc766e6387" candidate_uuid: "2d94505a8b6a4daab8fdbdffdf354596" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c60eee98017b49d4aa748b9ac4846cde"
I20260927 12:46:14.894688 2437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "561fae25418b4177be9143dc766e6387" candidate_uuid: "2d94505a8b6a4daab8fdbdffdf354596" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0060ce040449f1a4415d377932fa44"
I20260927 12:46:14.894745 2297 raft_consensus.cc:3037] T 561fae25418b4177be9143dc766e6387 P c60eee98017b49d4aa748b9ac4846cde [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:46:14.894807 2437 raft_consensus.cc:3037] T 561fae25418b4177be9143dc766e6387 P 1f0060ce040449f1a4415d377932fa44 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:46:14.895825 2437 raft_consensus.cc:2454] T 561fae25418b4177be9143dc766e6387 P 1f0060ce040449f1a4415d377932fa44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d94505a8b6a4daab8fdbdffdf354596 in term 1.
I20260927 12:46:14.895819 2297 raft_consensus.cc:2454] T 561fae25418b4177be9143dc766e6387 P c60eee98017b49d4aa748b9ac4846cde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d94505a8b6a4daab8fdbdffdf354596 in term 1.
I20260927 12:46:14.896133 2511 leader_election.cc:302] T 561fae25418b4177be9143dc766e6387 P 2d94505a8b6a4daab8fdbdffdf354596 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0060ce040449f1a4415d377932fa44, 2d94505a8b6a4daab8fdbdffdf354596; no voters:
I20260927 12:46:14.896347 2654 raft_consensus.cc:2781] T 561fae25418b4177be9143dc766e6387 P 2d94505a8b6a4daab8fdbdffdf354596 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:46:14.896517 2654 raft_consensus.cc:686] T 561fae25418b4177be9143dc766e6387 P 2d94505a8b6a4daab8fdbdffdf354596 [term 1 LEADER]: Becoming Leader. State: Replica: 2d94505a8b6a4daab8fdbdffdf354596, State: Running, Role: LEADER
I20260927 12:46:14.896890 2654 consensus_queue.cc:237] T 561fae25418b4177be9143dc766e6387 P 2d94505a8b6a4daab8fdbdffdf354596 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60eee98017b49d4aa748b9ac4846cde" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37687 } } peers { permanent_uuid: "1f0060ce040449f1a4415d377932fa44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45249 } } peers { permanent_uuid: "2d94505a8b6a4daab8fdbdffdf354596" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45131 } }
I20260927 12:46:14.897501 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:14.893011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34635 (local address 127.0.98.3:45131)
0927 12:46:14.893111 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:14.893115 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:14.893300 (+ 185us) server_negotiation.cc:407] Connection header received
0927 12:46:14.893468 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:14.893471 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:14.893509 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:14.893566 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:46:14.893794 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:14.894458 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:14.896071 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:14.896684 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:14.896750 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:14.896808 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:14.897295 (+ 487us) server_negotiation.cc:299] Negotiation successful
0927 12:46:14.897382 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 12:46:14.897805 2668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:14.892957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45131 (local address 127.0.98.1:34635)
0927 12:46:14.893227 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:14.893239 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:14.893365 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:14.893583 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:14.893585 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:14.893599 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:46:14.893706 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:14.893711 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:14.894546 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:14.894549 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:14.895951 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:14.895961 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:14.897045 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:14.897048 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:14.897159 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:14.897183 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 12:46:14.897606 (+ 423us) client_negotiation.cc:240] Negotiation successful
0927 12:46:14.897662 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":1,"thread_start_us":68,"threads_started":1}
I20260927 12:46:14.897811 2669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:14.893032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45249 (local address 127.0.98.1:34895)
0927 12:46:14.893280 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:14.893292 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:14.893384 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:14.893909 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:14.893912 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:14.893927 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:46:14.894477 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:14.894486 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:14.895429 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:14.895433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:14.896513 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:14.896522 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:14.897381 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:14.897384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:14.897473 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:14.897493 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 12:46:14.897563 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 12:46:14.897633 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":107,"threads_started":1}
I20260927 12:46:14.897959 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:14.893527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34895 (local address 127.0.98.2:45249)
0927 12:46:14.893641 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:14.893645 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:14.893657 (+ 12us) server_negotiation.cc:407] Connection header received
0927 12:46:14.893694 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:14.893697 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:14.893736 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:14.893815 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:46:14.894609 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:14.895313 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:14.896637 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:14.897185 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:14.897253 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:14.897305 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:14.897776 (+ 471us) server_negotiation.cc:299] Negotiation successful
0927 12:46:14.897851 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 12:46:14.898276 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "561fae25418b4177be9143dc766e6387" candidate_uuid: "c60eee98017b49d4aa748b9ac4846cde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d94505a8b6a4daab8fdbdffdf354596" is_pre_election: true
I20260927 12:46:14.898700 2437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "561fae25418b4177be9143dc766e6387" candidate_uuid: "c60eee98017b49d4aa748b9ac4846cde" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0060ce040449f1a4415d377932fa44" is_pre_election: true
I20260927 12:46:14.898892 2437 raft_consensus.cc:2379] T 561fae25418b4177be9143dc766e6387 P 1f0060ce040449f1a4415d377932fa44 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c60eee98017b49d4aa748b9ac4846cde in current term 1: Already voted for candidate 2d94505a8b6a4daab8fdbdffdf354596 in this term.
I20260927 12:46:14.899294 2231 leader_election.cc:302] T 561fae25418b4177be9143dc766e6387 P c60eee98017b49d4aa748b9ac4846cde [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c60eee98017b49d4aa748b9ac4846cde; no voters: 1f0060ce040449f1a4415d377932fa44, 2d94505a8b6a4daab8fdbdffdf354596
I20260927 12:46:14.899559 2652 raft_consensus.cc:2726] T 561fae25418b4177be9143dc766e6387 P c60eee98017b49d4aa748b9ac4846cde [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260927 12:46:14.900862 2670 tablet_replica.cc:439] T 561fae25418b4177be9143dc766e6387 P 2d94505a8b6a4daab8fdbdffdf354596: TxnStatusTablet state changed. Reason: New leader 2d94505a8b6a4daab8fdbdffdf354596. Latest consensus state: current_term: 1 leader_uuid: "2d94505a8b6a4daab8fdbdffdf354596" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60eee98017b49d4aa748b9ac4846cde" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37687 } } peers { permanent_uuid: "1f0060ce040449f1a4415d377932fa44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45249 } } peers { permanent_uuid: "2d94505a8b6a4daab8fdbdffdf354596" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45131 } } }
I20260927 12:46:14.901053 2670 tablet_replica.cc:442] T 561fae25418b4177be9143dc766e6387 P 2d94505a8b6a4daab8fdbdffdf354596: This TxnStatusTablet replica's current role is: LEADER
I20260927 12:46:14.901367 2672 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 12:46:14.901571 2672 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 12:46:14.901780 2137 catalog_manager.cc:5495] T 561fae25418b4177be9143dc766e6387 P 2d94505a8b6a4daab8fdbdffdf354596 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d94505a8b6a4daab8fdbdffdf354596 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "2d94505a8b6a4daab8fdbdffdf354596" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60eee98017b49d4aa748b9ac4846cde" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f0060ce040449f1a4415d377932fa44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d94505a8b6a4daab8fdbdffdf354596" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45131 } health_report { overall_health: HEALTHY } } }
I20260927 12:46:14.966003 2663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:14.962412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45131 (local address 127.0.0.1:48882)
0927 12:46:14.962552 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:14.962568 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:14.962672 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:14.962962 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:14.962965 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:14.962971 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 12:46:14.963073 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:14.963079 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:14.963781 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:14.963785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:14.964400 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:14.964408 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:14.964834 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:14.964837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:14.964931 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:14.964980 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 12:46:14.965690 (+ 710us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 12:46:14.965708 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 12:46:14.965782 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 12:46:14.965882 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260927 12:46:14.966027 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:14.962535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48882 (local address 127.0.98.3:45131)
0927 12:46:14.962654 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:14.962658 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:14.962673 (+ 15us) server_negotiation.cc:407] Connection header received
0927 12:46:14.962804 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:14.962807 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:14.962862 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:14.962940 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:46:14.963165 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:14.963674 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:14.964509 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:14.964724 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:14.964833 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:14.965115 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:46:14.965596 (+ 481us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:46:14.965719 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:14.965800 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 12:46:14.965855 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 12:46:14.971210 2437 raft_consensus.cc:1261] T 561fae25418b4177be9143dc766e6387 P 1f0060ce040449f1a4415d377932fa44 [term 1 FOLLOWER]: Refusing update from remote peer 2d94505a8b6a4daab8fdbdffdf354596: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:46:14.971452 2297 raft_consensus.cc:1261] T 561fae25418b4177be9143dc766e6387 P c60eee98017b49d4aa748b9ac4846cde [term 1 FOLLOWER]: Refusing update from remote peer 2d94505a8b6a4daab8fdbdffdf354596: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:46:14.971848 2670 consensus_queue.cc:1035] T 561fae25418b4177be9143dc766e6387 P 2d94505a8b6a4daab8fdbdffdf354596 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0060ce040449f1a4415d377932fa44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:46:14.972563 2670 consensus_queue.cc:1035] T 561fae25418b4177be9143dc766e6387 P 2d94505a8b6a4daab8fdbdffdf354596 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c60eee98017b49d4aa748b9ac4846cde" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:46:14.977059 2653 tablet_replica.cc:439] T 561fae25418b4177be9143dc766e6387 P 1f0060ce040449f1a4415d377932fa44: TxnStatusTablet state changed. Reason: New leader 2d94505a8b6a4daab8fdbdffdf354596. Latest consensus state: current_term: 1 leader_uuid: "2d94505a8b6a4daab8fdbdffdf354596" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60eee98017b49d4aa748b9ac4846cde" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37687 } } peers { permanent_uuid: "1f0060ce040449f1a4415d377932fa44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45249 } } peers { permanent_uuid: "2d94505a8b6a4daab8fdbdffdf354596" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45131 } } }
I20260927 12:46:14.977195 2653 tablet_replica.cc:442] T 561fae25418b4177be9143dc766e6387 P 1f0060ce040449f1a4415d377932fa44: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 12:46:14.978181 2652 tablet_replica.cc:439] T 561fae25418b4177be9143dc766e6387 P c60eee98017b49d4aa748b9ac4846cde: TxnStatusTablet state changed. Reason: New leader 2d94505a8b6a4daab8fdbdffdf354596. Latest consensus state: current_term: 1 leader_uuid: "2d94505a8b6a4daab8fdbdffdf354596" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60eee98017b49d4aa748b9ac4846cde" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37687 } } peers { permanent_uuid: "1f0060ce040449f1a4415d377932fa44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45249 } } peers { permanent_uuid: "2d94505a8b6a4daab8fdbdffdf354596" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45131 } } }
I20260927 12:46:14.978331 2652 tablet_replica.cc:442] T 561fae25418b4177be9143dc766e6387 P c60eee98017b49d4aa748b9ac4846cde: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 12:46:14.979691 2675 mvcc.cc:205] Tried to move back new op lower bound from 7333941964675198976 to 7333941964380483584. Current Snapshot: MvccSnapshot[applied={T|T < 7333941964675198976}]
I20260927 12:46:14.980149 2654 tablet_replica.cc:439] T 561fae25418b4177be9143dc766e6387 P 2d94505a8b6a4daab8fdbdffdf354596: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2d94505a8b6a4daab8fdbdffdf354596" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60eee98017b49d4aa748b9ac4846cde" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37687 } } peers { permanent_uuid: "1f0060ce040449f1a4415d377932fa44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45249 } } peers { permanent_uuid: "2d94505a8b6a4daab8fdbdffdf354596" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45131 } } }
I20260927 12:46:14.980260 2654 tablet_replica.cc:442] T 561fae25418b4177be9143dc766e6387 P 2d94505a8b6a4daab8fdbdffdf354596: This TxnStatusTablet replica's current role is: LEADER
I20260927 12:46:14.980369 2676 mvcc.cc:205] Tried to move back new op lower bound from 7333941964675198976 to 7333941964380483584. Current Snapshot: MvccSnapshot[applied={T|T < 7333941964675198976}]
I20260927 12:46:14.980640 2653 tablet_replica.cc:439] T 561fae25418b4177be9143dc766e6387 P 1f0060ce040449f1a4415d377932fa44: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2d94505a8b6a4daab8fdbdffdf354596" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60eee98017b49d4aa748b9ac4846cde" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37687 } } peers { permanent_uuid: "1f0060ce040449f1a4415d377932fa44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45249 } } peers { permanent_uuid: "2d94505a8b6a4daab8fdbdffdf354596" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45131 } } }
I20260927 12:46:14.980744 2653 tablet_replica.cc:442] T 561fae25418b4177be9143dc766e6387 P 1f0060ce040449f1a4415d377932fa44: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 12:46:14.984421 2670 tablet_replica.cc:439] T 561fae25418b4177be9143dc766e6387 P 2d94505a8b6a4daab8fdbdffdf354596: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2d94505a8b6a4daab8fdbdffdf354596" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60eee98017b49d4aa748b9ac4846cde" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37687 } } peers { permanent_uuid: "1f0060ce040449f1a4415d377932fa44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45249 } } peers { permanent_uuid: "2d94505a8b6a4daab8fdbdffdf354596" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45131 } } }
I20260927 12:46:14.984545 2670 tablet_replica.cc:442] T 561fae25418b4177be9143dc766e6387 P 2d94505a8b6a4daab8fdbdffdf354596: This TxnStatusTablet replica's current role is: LEADER
I20260927 12:46:14.985703 2678 mvcc.cc:205] Tried to move back new op lower bound from 7333941964675198976 to 7333941964380483584. Current Snapshot: MvccSnapshot[applied={T|T < 7333941964675198976}]
I20260927 12:46:14.985819 2652 tablet_replica.cc:439] T 561fae25418b4177be9143dc766e6387 P c60eee98017b49d4aa748b9ac4846cde: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2d94505a8b6a4daab8fdbdffdf354596" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60eee98017b49d4aa748b9ac4846cde" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37687 } } peers { permanent_uuid: "1f0060ce040449f1a4415d377932fa44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45249 } } peers { permanent_uuid: "2d94505a8b6a4daab8fdbdffdf354596" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45131 } } }
I20260927 12:46:14.985932 2652 tablet_replica.cc:442] T 561fae25418b4177be9143dc766e6387 P c60eee98017b49d4aa748b9ac4846cde: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 12:46:14.990258 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:14.986809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36386 (local address 127.0.98.2:45249)
0927 12:46:14.986923 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:14.986927 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:14.986941 (+ 14us) server_negotiation.cc:407] Connection header received
0927 12:46:14.987025 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:14.987028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:14.987085 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:14.987194 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:46:14.987465 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:14.988018 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:14.988862 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:14.989089 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:14.989174 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:14.989451 (+ 277us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:46:14.989855 (+ 404us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:46:14.989952 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:14.990067 (+ 115us) server_negotiation.cc:299] Negotiation successful
0927 12:46:14.990124 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 12:46:15.002311 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:14.998613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40755 (local address 127.0.98.3:45131)
0927 12:46:14.998720 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:14.998724 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:14.998976 (+ 252us) server_negotiation.cc:407] Connection header received
0927 12:46:14.999106 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:14.999108 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:14.999151 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:14.999211 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:46:14.999467 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:14.999992 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:15.001217 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:15.001439 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:15.001516 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:15.001869 (+ 353us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:46:15.001950 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:46:15.001995 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:15.002127 (+ 132us) server_negotiation.cc:299] Negotiation successful
0927 12:46:15.002175 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 12:46:15.002377 2687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:14.998548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45131 (local address 127.0.98.2:40755)
0927 12:46:14.998900 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:14.998914 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:14.999021 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:14.999233 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:14.999236 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:14.999244 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 12:46:14.999381 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:14.999387 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:15.000086 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:15.000090 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:15.001098 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:15.001110 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:15.001553 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:15.001557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:15.001649 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:15.001707 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 12:46:15.002018 (+ 311us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 12:46:15.002023 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 12:46:15.002119 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 12:46:15.002205 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":86,"threads_started":1}
I20260927 12:46:15.009850 2297 raft_consensus.cc:1261] T d897625dc49c462fb17b027944540640 P c60eee98017b49d4aa748b9ac4846cde [term 1 FOLLOWER]: Refusing update from remote peer 1f0060ce040449f1a4415d377932fa44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:46:15.009888 2577 raft_consensus.cc:1261] T d897625dc49c462fb17b027944540640 P 2d94505a8b6a4daab8fdbdffdf354596 [term 1 FOLLOWER]: Refusing update from remote peer 1f0060ce040449f1a4415d377932fa44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:46:15.010339 2653 consensus_queue.cc:1035] T d897625dc49c462fb17b027944540640 P 1f0060ce040449f1a4415d377932fa44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d94505a8b6a4daab8fdbdffdf354596" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:46:15.010520 2656 consensus_queue.cc:1035] T d897625dc49c462fb17b027944540640 P 1f0060ce040449f1a4415d377932fa44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c60eee98017b49d4aa748b9ac4846cde" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:46:15.034773 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:15.030787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45249 (local address 127.0.98.3:51681)
0927 12:46:15.030941 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:15.030956 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:15.031055 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:15.031403 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:15.031406 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:15.031413 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 12:46:15.031527 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:15.031533 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:15.032535 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:15.032538 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:15.033495 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:15.033506 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:15.033949 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:15.033953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:15.034081 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:15.034140 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 12:46:15.034460 (+ 320us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 12:46:15.034467 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 12:46:15.034536 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 12:46:15.034615 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 12:46:15.034789 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:15.030746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51681 (local address 127.0.98.2:45249)
0927 12:46:15.030868 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:15.030873 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:15.031023 (+ 150us) server_negotiation.cc:407] Connection header received
0927 12:46:15.031175 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:15.031178 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:15.031225 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:15.031296 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:46:15.031635 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:15.032405 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:15.033622 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:15.033840 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:15.033921 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:15.034302 (+ 381us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:46:15.034375 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:46:15.034472 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:15.034545 (+ 73us) server_negotiation.cc:299] Negotiation successful
0927 12:46:15.034591 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 12:46:15.063647 2137 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:46:15.063882 2137 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:46:15.066656 2137 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 92ba4aec5bf34fde9f6049baedcb6f3f: Sending DeleteTablet for 3 replicas of tablet d897625dc49c462fb17b027944540640
I20260927 12:46:15.067426 2417 tablet_service.cc:1514] Processing DeleteTablet for tablet d897625dc49c462fb17b027944540640 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:46:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36378
I20260927 12:46:15.067528 2277 tablet_service.cc:1514] Processing DeleteTablet for tablet d897625dc49c462fb17b027944540640 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:46:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39256
I20260927 12:46:15.067819 2557 tablet_service.cc:1514] Processing DeleteTablet for tablet d897625dc49c462fb17b027944540640 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:46:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48880
I20260927 12:46:15.067951 2694 tablet_replica.cc:330] T d897625dc49c462fb17b027944540640 P 1f0060ce040449f1a4415d377932fa44: stopping tablet replica
I20260927 12:46:15.068228 2696 tablet_replica.cc:330] T d897625dc49c462fb17b027944540640 P 2d94505a8b6a4daab8fdbdffdf354596: stopping tablet replica
I20260927 12:46:15.068266 2694 raft_consensus.cc:2229] T d897625dc49c462fb17b027944540640 P 1f0060ce040449f1a4415d377932fa44 [term 1 LEADER]: Raft consensus shutting down.
I20260927 12:46:15.068630 2694 raft_consensus.cc:2258] T d897625dc49c462fb17b027944540640 P 1f0060ce040449f1a4415d377932fa44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:46:15.069144 2696 raft_consensus.cc:2229] T d897625dc49c462fb17b027944540640 P 2d94505a8b6a4daab8fdbdffdf354596 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:46:15.069312 2696 raft_consensus.cc:2258] T d897625dc49c462fb17b027944540640 P 2d94505a8b6a4daab8fdbdffdf354596 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:46:15.069413 2137 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55234:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:46:15.069995 2137 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:46:15.070165 2694 ts_tablet_manager.cc:1905] T d897625dc49c462fb17b027944540640 P 1f0060ce040449f1a4415d377932fa44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:46:15.070585 2696 ts_tablet_manager.cc:1905] T d897625dc49c462fb17b027944540640 P 2d94505a8b6a4daab8fdbdffdf354596: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:46:15.073228 2694 ts_tablet_manager.cc:1918] T d897625dc49c462fb17b027944540640 P 1f0060ce040449f1a4415d377932fa44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 12:46:15.073333 2694 log.cc:1196] T d897625dc49c462fb17b027944540640 P 1f0060ce040449f1a4415d377932fa44: Deleting WAL directory at /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/ts-1/wal/wals/d897625dc49c462fb17b027944540640
I20260927 12:46:15.073633 2694 ts_tablet_manager.cc:1939] T d897625dc49c462fb17b027944540640 P 1f0060ce040449f1a4415d377932fa44: Deleting consensus metadata
I20260927 12:46:15.075081 2417 tablet_service.cc:1467] Processing CreateTablet for tablet 94419de0777d47ea86a250fc3d8978c5 (DEFAULT_TABLE table=test-table [id=d2582fff44fc44359d95cc02f657a1ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:46:15.075104 2277 tablet_service.cc:1467] Processing CreateTablet for tablet 94419de0777d47ea86a250fc3d8978c5 (DEFAULT_TABLE table=test-table [id=d2582fff44fc44359d95cc02f657a1ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:46:15.075347 2417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94419de0777d47ea86a250fc3d8978c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:46:15.075366 2277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94419de0777d47ea86a250fc3d8978c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:46:15.075336 2557 tablet_service.cc:1467] Processing CreateTablet for tablet 94419de0777d47ea86a250fc3d8978c5 (DEFAULT_TABLE table=test-table [id=d2582fff44fc44359d95cc02f657a1ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:46:15.075657 2557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94419de0777d47ea86a250fc3d8978c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:46:15.077543 2647 tablet_bootstrap.cc:492] T 94419de0777d47ea86a250fc3d8978c5 P 1f0060ce040449f1a4415d377932fa44: Bootstrap starting.
I20260927 12:46:15.077746 2646 tablet_bootstrap.cc:492] T 94419de0777d47ea86a250fc3d8978c5 P 2d94505a8b6a4daab8fdbdffdf354596: Bootstrap starting.
I20260927 12:46:15.077806 2648 tablet_bootstrap.cc:492] T 94419de0777d47ea86a250fc3d8978c5 P c60eee98017b49d4aa748b9ac4846cde: Bootstrap starting.
I20260927 12:46:15.078500 2646 tablet_bootstrap.cc:654] T 94419de0777d47ea86a250fc3d8978c5 P 2d94505a8b6a4daab8fdbdffdf354596: Neither blocks nor log segments found. Creating new log.
I20260927 12:46:15.078500 2647 tablet_bootstrap.cc:654] T 94419de0777d47ea86a250fc3d8978c5 P 1f0060ce040449f1a4415d377932fa44: Neither blocks nor log segments found. Creating new log.
I20260927 12:46:15.078701 2648 tablet_bootstrap.cc:654] T 94419de0777d47ea86a250fc3d8978c5 P c60eee98017b49d4aa748b9ac4846cde: Neither blocks nor log segments found. Creating new log.
I20260927 12:46:15.079417 2127 catalog_manager.cc:4841] TS 1f0060ce040449f1a4415d377932fa44 (127.0.98.2:45249): tablet d897625dc49c462fb17b027944540640 (table test-table [id=65a1e66cd92842d29f2c0496ad5e1b07]) successfully deleted
I20260927 12:46:15.079761 2696 ts_tablet_manager.cc:1918] T d897625dc49c462fb17b027944540640 P 2d94505a8b6a4daab8fdbdffdf354596: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 12:46:15.079809 2695 tablet_replica.cc:330] T d897625dc49c462fb17b027944540640 P c60eee98017b49d4aa748b9ac4846cde: stopping tablet replica
I20260927 12:46:15.079881 2696 log.cc:1196] T d897625dc49c462fb17b027944540640 P 2d94505a8b6a4daab8fdbdffdf354596: Deleting WAL directory at /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/ts-2/wal/wals/d897625dc49c462fb17b027944540640
I20260927 12:46:15.080150 2695 raft_consensus.cc:2229] T d897625dc49c462fb17b027944540640 P c60eee98017b49d4aa748b9ac4846cde [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:46:15.080288 2647 tablet_bootstrap.cc:492] T 94419de0777d47ea86a250fc3d8978c5 P 1f0060ce040449f1a4415d377932fa44: No bootstrap required, opened a new log
I20260927 12:46:15.080178 2696 ts_tablet_manager.cc:1939] T d897625dc49c462fb17b027944540640 P 2d94505a8b6a4daab8fdbdffdf354596: Deleting consensus metadata
I20260927 12:46:15.080367 2647 ts_tablet_manager.cc:1397] T 94419de0777d47ea86a250fc3d8978c5 P 1f0060ce040449f1a4415d377932fa44: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 12:46:15.080416 2695 raft_consensus.cc:2258] T d897625dc49c462fb17b027944540640 P c60eee98017b49d4aa748b9ac4846cde [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:46:15.080797 2647 raft_consensus.cc:348] T 94419de0777d47ea86a250fc3d8978c5 P 1f0060ce040449f1a4415d377932fa44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60eee98017b49d4aa748b9ac4846cde" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37687 } } peers { permanent_uuid: "1f0060ce040449f1a4415d377932fa44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45249 } } peers { permanent_uuid: "2d94505a8b6a4daab8fdbdffdf354596" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45131 } }
I20260927 12:46:15.080919 2647 raft_consensus.cc:374] T 94419de0777d47ea86a250fc3d8978c5 P 1f0060ce040449f1a4415d377932fa44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:46:15.080947 2647 raft_consensus.cc:729] T 94419de0777d47ea86a250fc3d8978c5 P 1f0060ce040449f1a4415d377932fa44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0060ce040449f1a4415d377932fa44, State: Initialized, Role: FOLLOWER
I20260927 12:46:15.081063 2647 consensus_queue.cc:260] T 94419de0777d47ea86a250fc3d8978c5 P 1f0060ce040449f1a4415d377932fa44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60eee98017b49d4aa748b9ac4846cde" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37687 } } peers { permanent_uuid: "1f0060ce040449f1a4415d377932fa44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45249 } } peers { permanent_uuid: "2d94505a8b6a4daab8fdbdffdf354596" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45131 } }
I20260927 12:46:15.081302 2647 ts_tablet_manager.cc:1428] T 94419de0777d47ea86a250fc3d8978c5 P 1f0060ce040449f1a4415d377932fa44: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 12:46:15.081583 2124 catalog_manager.cc:4841] TS 2d94505a8b6a4daab8fdbdffdf354596 (127.0.98.3:45131): tablet d897625dc49c462fb17b027944540640 (table test-table [id=65a1e66cd92842d29f2c0496ad5e1b07]) successfully deleted
I20260927 12:46:15.082521 2646 tablet_bootstrap.cc:492] T 94419de0777d47ea86a250fc3d8978c5 P 2d94505a8b6a4daab8fdbdffdf354596: No bootstrap required, opened a new log
I20260927 12:46:15.082610 2646 ts_tablet_manager.cc:1397] T 94419de0777d47ea86a250fc3d8978c5 P 2d94505a8b6a4daab8fdbdffdf354596: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260927 12:46:15.083009 2646 raft_consensus.cc:348] T 94419de0777d47ea86a250fc3d8978c5 P 2d94505a8b6a4daab8fdbdffdf354596 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60eee98017b49d4aa748b9ac4846cde" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37687 } } peers { permanent_uuid: "1f0060ce040449f1a4415d377932fa44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45249 } } peers { permanent_uuid: "2d94505a8b6a4daab8fdbdffdf354596" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45131 } }
I20260927 12:46:15.083118 2646 raft_consensus.cc:374] T 94419de0777d47ea86a250fc3d8978c5 P 2d94505a8b6a4daab8fdbdffdf354596 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:46:15.083148 2646 raft_consensus.cc:729] T 94419de0777d47ea86a250fc3d8978c5 P 2d94505a8b6a4daab8fdbdffdf354596 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d94505a8b6a4daab8fdbdffdf354596, State: Initialized, Role: FOLLOWER
I20260927 12:46:15.083264 2646 consensus_queue.cc:260] T 94419de0777d47ea86a250fc3d8978c5 P 2d94505a8b6a4daab8fdbdffdf354596 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60eee98017b49d4aa748b9ac4846cde" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37687 } } peers { permanent_uuid: "1f0060ce040449f1a4415d377932fa44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45249 } } peers { permanent_uuid: "2d94505a8b6a4daab8fdbdffdf354596" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45131 } }
I20260927 12:46:15.083518 2646 ts_tablet_manager.cc:1428] T 94419de0777d47ea86a250fc3d8978c5 P 2d94505a8b6a4daab8fdbdffdf354596: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 12:46:15.087080 2648 tablet_bootstrap.cc:492] T 94419de0777d47ea86a250fc3d8978c5 P c60eee98017b49d4aa748b9ac4846cde: No bootstrap required, opened a new log
I20260927 12:46:15.087169 2648 ts_tablet_manager.cc:1397] T 94419de0777d47ea86a250fc3d8978c5 P c60eee98017b49d4aa748b9ac4846cde: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20260927 12:46:15.087338 2695 ts_tablet_manager.cc:1905] T d897625dc49c462fb17b027944540640 P c60eee98017b49d4aa748b9ac4846cde: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:46:15.087572 2648 raft_consensus.cc:348] T 94419de0777d47ea86a250fc3d8978c5 P c60eee98017b49d4aa748b9ac4846cde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60eee98017b49d4aa748b9ac4846cde" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37687 } } peers { permanent_uuid: "1f0060ce040449f1a4415d377932fa44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45249 } } peers { permanent_uuid: "2d94505a8b6a4daab8fdbdffdf354596" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45131 } }
I20260927 12:46:15.087678 2648 raft_consensus.cc:374] T 94419de0777d47ea86a250fc3d8978c5 P c60eee98017b49d4aa748b9ac4846cde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:46:15.087705 2648 raft_consensus.cc:729] T 94419de0777d47ea86a250fc3d8978c5 P c60eee98017b49d4aa748b9ac4846cde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c60eee98017b49d4aa748b9ac4846cde, State: Initialized, Role: FOLLOWER
I20260927 12:46:15.087832 2648 consensus_queue.cc:260] T 94419de0777d47ea86a250fc3d8978c5 P c60eee98017b49d4aa748b9ac4846cde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60eee98017b49d4aa748b9ac4846cde" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37687 } } peers { permanent_uuid: "1f0060ce040449f1a4415d377932fa44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45249 } } peers { permanent_uuid: "2d94505a8b6a4daab8fdbdffdf354596" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45131 } }
I20260927 12:46:15.088078 2648 ts_tablet_manager.cc:1428] T 94419de0777d47ea86a250fc3d8978c5 P c60eee98017b49d4aa748b9ac4846cde: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 12:46:15.090247 2695 ts_tablet_manager.cc:1918] T d897625dc49c462fb17b027944540640 P c60eee98017b49d4aa748b9ac4846cde: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 12:46:15.090354 2695 log.cc:1196] T d897625dc49c462fb17b027944540640 P c60eee98017b49d4aa748b9ac4846cde: Deleting WAL directory at /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/ts-0/wal/wals/d897625dc49c462fb17b027944540640
I20260927 12:46:15.090646 2695 ts_tablet_manager.cc:1939] T d897625dc49c462fb17b027944540640 P c60eee98017b49d4aa748b9ac4846cde: Deleting consensus metadata
I20260927 12:46:15.092046 2126 catalog_manager.cc:4841] TS c60eee98017b49d4aa748b9ac4846cde (127.0.98.1:37687): tablet d897625dc49c462fb17b027944540640 (table test-table [id=65a1e66cd92842d29f2c0496ad5e1b07]) successfully deleted
I20260927 12:46:15.126852 2682 raft_consensus.cc:482] T 94419de0777d47ea86a250fc3d8978c5 P 2d94505a8b6a4daab8fdbdffdf354596 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:46:15.127007 2682 raft_consensus.cc:504] T 94419de0777d47ea86a250fc3d8978c5 P 2d94505a8b6a4daab8fdbdffdf354596 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60eee98017b49d4aa748b9ac4846cde" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37687 } } peers { permanent_uuid: "1f0060ce040449f1a4415d377932fa44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45249 } } peers { permanent_uuid: "2d94505a8b6a4daab8fdbdffdf354596" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45131 } }
I20260927 12:46:15.127477 2682 leader_election.cc:288] T 94419de0777d47ea86a250fc3d8978c5 P 2d94505a8b6a4daab8fdbdffdf354596 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c60eee98017b49d4aa748b9ac4846cde (127.0.98.1:37687), 1f0060ce040449f1a4415d377932fa44 (127.0.98.2:45249)
I20260927 12:46:15.127961 2437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94419de0777d47ea86a250fc3d8978c5" candidate_uuid: "2d94505a8b6a4daab8fdbdffdf354596" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0060ce040449f1a4415d377932fa44" is_pre_election: true
I20260927 12:46:15.128028 2297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94419de0777d47ea86a250fc3d8978c5" candidate_uuid: "2d94505a8b6a4daab8fdbdffdf354596" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c60eee98017b49d4aa748b9ac4846cde" is_pre_election: true
I20260927 12:46:15.128108 2437 raft_consensus.cc:2454] T 94419de0777d47ea86a250fc3d8978c5 P 1f0060ce040449f1a4415d377932fa44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d94505a8b6a4daab8fdbdffdf354596 in term 0.
I20260927 12:46:15.128222 2297 raft_consensus.cc:2454] T 94419de0777d47ea86a250fc3d8978c5 P c60eee98017b49d4aa748b9ac4846cde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d94505a8b6a4daab8fdbdffdf354596 in term 0.
I20260927 12:46:15.128544 2511 leader_election.cc:302] T 94419de0777d47ea86a250fc3d8978c5 P 2d94505a8b6a4daab8fdbdffdf354596 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0060ce040449f1a4415d377932fa44, 2d94505a8b6a4daab8fdbdffdf354596; no voters:
I20260927 12:46:15.128746 2682 raft_consensus.cc:2781] T 94419de0777d47ea86a250fc3d8978c5 P 2d94505a8b6a4daab8fdbdffdf354596 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:46:15.128809 2682 raft_consensus.cc:482] T 94419de0777d47ea86a250fc3d8978c5 P 2d94505a8b6a4daab8fdbdffdf354596 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:46:15.128849 2682 raft_consensus.cc:3037] T 94419de0777d47ea86a250fc3d8978c5 P 2d94505a8b6a4daab8fdbdffdf354596 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:46:15.130067 2682 raft_consensus.cc:504] T 94419de0777d47ea86a250fc3d8978c5 P 2d94505a8b6a4daab8fdbdffdf354596 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60eee98017b49d4aa748b9ac4846cde" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37687 } } peers { permanent_uuid: "1f0060ce040449f1a4415d377932fa44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45249 } } peers { permanent_uuid: "2d94505a8b6a4daab8fdbdffdf354596" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45131 } }
I20260927 12:46:15.130443 2682 leader_election.cc:288] T 94419de0777d47ea86a250fc3d8978c5 P 2d94505a8b6a4daab8fdbdffdf354596 [CANDIDATE]: Term 1 election: Requested vote from peers c60eee98017b49d4aa748b9ac4846cde (127.0.98.1:37687), 1f0060ce040449f1a4415d377932fa44 (127.0.98.2:45249)
I20260927 12:46:15.130808 2437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94419de0777d47ea86a250fc3d8978c5" candidate_uuid: "2d94505a8b6a4daab8fdbdffdf354596" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f0060ce040449f1a4415d377932fa44"
I20260927 12:46:15.130869 2297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94419de0777d47ea86a250fc3d8978c5" candidate_uuid: "2d94505a8b6a4daab8fdbdffdf354596" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c60eee98017b49d4aa748b9ac4846cde"
I20260927 12:46:15.130941 2437 raft_consensus.cc:3037] T 94419de0777d47ea86a250fc3d8978c5 P 1f0060ce040449f1a4415d377932fa44 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:46:15.131006 2297 raft_consensus.cc:3037] T 94419de0777d47ea86a250fc3d8978c5 P c60eee98017b49d4aa748b9ac4846cde [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:46:15.131903 2437 raft_consensus.cc:2454] T 94419de0777d47ea86a250fc3d8978c5 P 1f0060ce040449f1a4415d377932fa44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d94505a8b6a4daab8fdbdffdf354596 in term 1.
I20260927 12:46:15.132102 2297 raft_consensus.cc:2454] T 94419de0777d47ea86a250fc3d8978c5 P c60eee98017b49d4aa748b9ac4846cde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d94505a8b6a4daab8fdbdffdf354596 in term 1.
I20260927 12:46:15.132189 2511 leader_election.cc:302] T 94419de0777d47ea86a250fc3d8978c5 P 2d94505a8b6a4daab8fdbdffdf354596 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0060ce040449f1a4415d377932fa44, 2d94505a8b6a4daab8fdbdffdf354596; no voters:
I20260927 12:46:15.132372 2682 raft_consensus.cc:2781] T 94419de0777d47ea86a250fc3d8978c5 P 2d94505a8b6a4daab8fdbdffdf354596 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:46:15.132463 2682 raft_consensus.cc:686] T 94419de0777d47ea86a250fc3d8978c5 P 2d94505a8b6a4daab8fdbdffdf354596 [term 1 LEADER]: Becoming Leader. State: Replica: 2d94505a8b6a4daab8fdbdffdf354596, State: Running, Role: LEADER
I20260927 12:46:15.132601 2682 consensus_queue.cc:237] T 94419de0777d47ea86a250fc3d8978c5 P 2d94505a8b6a4daab8fdbdffdf354596 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60eee98017b49d4aa748b9ac4846cde" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37687 } } peers { permanent_uuid: "1f0060ce040449f1a4415d377932fa44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45249 } } peers { permanent_uuid: "2d94505a8b6a4daab8fdbdffdf354596" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45131 } }
I20260927 12:46:15.134559 2137 catalog_manager.cc:5495] T 94419de0777d47ea86a250fc3d8978c5 P 2d94505a8b6a4daab8fdbdffdf354596 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d94505a8b6a4daab8fdbdffdf354596 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "2d94505a8b6a4daab8fdbdffdf354596" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60eee98017b49d4aa748b9ac4846cde" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f0060ce040449f1a4415d377932fa44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d94505a8b6a4daab8fdbdffdf354596" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45131 } health_report { overall_health: HEALTHY } } }
I20260927 12:46:15.156258 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:15.152874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48896 (local address 127.0.98.3:45131)
0927 12:46:15.152998 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:15.153003 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:15.153017 (+ 14us) server_negotiation.cc:407] Connection header received
0927 12:46:15.153139 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:15.153142 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:15.153193 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:15.153278 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:46:15.153486 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:15.154307 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:15.155115 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:15.155454 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:15.155554 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:15.155771 (+ 217us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:46:15.155855 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:46:15.155918 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:15.156054 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 12:46:15.156108 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 12:46:15.165606 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:15.162689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45131 (local address 127.0.98.3:43205)
0927 12:46:15.162816 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:15.162833 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:15.162929 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:15.163110 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:15.163113 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:15.163118 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 12:46:15.163229 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:15.163234 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:15.163933 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:15.163936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:15.164517 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:15.164525 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:15.164922 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:15.164925 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:15.165023 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:15.165045 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 12:46:15.165284 (+ 239us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 12:46:15.165290 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 12:46:15.165372 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 12:46:15.165456 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":35}
I20260927 12:46:15.165609 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:15.162761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43205 (local address 127.0.98.3:45131)
0927 12:46:15.162918 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:15.162922 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:15.162935 (+ 13us) server_negotiation.cc:407] Connection header received
0927 12:46:15.162998 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:15.163001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:15.163040 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:15.163094 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:46:15.163355 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:15.163843 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:15.164618 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:15.164830 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:15.164911 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:15.165140 (+ 229us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:46:15.165207 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:46:15.165267 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:15.165414 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 12:46:15.165474 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":103}
I20260927 12:46:15.171691 2437 raft_consensus.cc:1261] T 94419de0777d47ea86a250fc3d8978c5 P 1f0060ce040449f1a4415d377932fa44 [term 1 FOLLOWER]: Refusing update from remote peer 2d94505a8b6a4daab8fdbdffdf354596: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:46:15.172176 2297 raft_consensus.cc:1261] T 94419de0777d47ea86a250fc3d8978c5 P c60eee98017b49d4aa748b9ac4846cde [term 1 FOLLOWER]: Refusing update from remote peer 2d94505a8b6a4daab8fdbdffdf354596: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:46:15.172220 2682 consensus_queue.cc:1035] T 94419de0777d47ea86a250fc3d8978c5 P 2d94505a8b6a4daab8fdbdffdf354596 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f0060ce040449f1a4415d377932fa44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:46:15.172590 2682 consensus_queue.cc:1035] T 94419de0777d47ea86a250fc3d8978c5 P 2d94505a8b6a4daab8fdbdffdf354596 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c60eee98017b49d4aa748b9ac4846cde" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 12:46:15.187249 2675 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 12:46:16.745267 2139 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:46:16.745424 2139 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:46:16.748071 2139 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 92ba4aec5bf34fde9f6049baedcb6f3f: Sending DeleteTablet for 3 replicas of tablet 94419de0777d47ea86a250fc3d8978c5
I20260927 12:46:16.748732 2417 tablet_service.cc:1514] Processing DeleteTablet for tablet 94419de0777d47ea86a250fc3d8978c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:46:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36378
I20260927 12:46:16.748733 2557 tablet_service.cc:1514] Processing DeleteTablet for tablet 94419de0777d47ea86a250fc3d8978c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:46:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48880
I20260927 12:46:16.748776 2277 tablet_service.cc:1514] Processing DeleteTablet for tablet 94419de0777d47ea86a250fc3d8978c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:46:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39256
I20260927 12:46:16.749173 2726 tablet_replica.cc:330] T 94419de0777d47ea86a250fc3d8978c5 P 2d94505a8b6a4daab8fdbdffdf354596: stopping tablet replica
I20260927 12:46:16.749259 2727 tablet_replica.cc:330] T 94419de0777d47ea86a250fc3d8978c5 P c60eee98017b49d4aa748b9ac4846cde: stopping tablet replica
I20260927 12:46:16.749325 2726 raft_consensus.cc:2229] T 94419de0777d47ea86a250fc3d8978c5 P 2d94505a8b6a4daab8fdbdffdf354596 [term 1 LEADER]: Raft consensus shutting down.
I20260927 12:46:16.749372 2727 raft_consensus.cc:2229] T 94419de0777d47ea86a250fc3d8978c5 P c60eee98017b49d4aa748b9ac4846cde [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:46:16.749485 2727 raft_consensus.cc:2258] T 94419de0777d47ea86a250fc3d8978c5 P c60eee98017b49d4aa748b9ac4846cde [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:46:16.749662 2726 raft_consensus.cc:2258] T 94419de0777d47ea86a250fc3d8978c5 P 2d94505a8b6a4daab8fdbdffdf354596 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:46:16.749900 2725 tablet_replica.cc:330] T 94419de0777d47ea86a250fc3d8978c5 P 1f0060ce040449f1a4415d377932fa44: stopping tablet replica
I20260927 12:46:16.750026 2725 raft_consensus.cc:2229] T 94419de0777d47ea86a250fc3d8978c5 P 1f0060ce040449f1a4415d377932fa44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:46:16.750173 2725 raft_consensus.cc:2258] T 94419de0777d47ea86a250fc3d8978c5 P 1f0060ce040449f1a4415d377932fa44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:46:16.750301 2727 ts_tablet_manager.cc:1905] T 94419de0777d47ea86a250fc3d8978c5 P c60eee98017b49d4aa748b9ac4846cde: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:46:16.750797 2726 ts_tablet_manager.cc:1905] T 94419de0777d47ea86a250fc3d8978c5 P 2d94505a8b6a4daab8fdbdffdf354596: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:46:16.751031 2725 ts_tablet_manager.cc:1905] T 94419de0777d47ea86a250fc3d8978c5 P 1f0060ce040449f1a4415d377932fa44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:46:16.753044 2727 ts_tablet_manager.cc:1918] T 94419de0777d47ea86a250fc3d8978c5 P c60eee98017b49d4aa748b9ac4846cde: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 12:46:16.753131 2727 log.cc:1196] T 94419de0777d47ea86a250fc3d8978c5 P c60eee98017b49d4aa748b9ac4846cde: Deleting WAL directory at /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/ts-0/wal/wals/94419de0777d47ea86a250fc3d8978c5
I20260927 12:46:16.753330 2726 ts_tablet_manager.cc:1918] T 94419de0777d47ea86a250fc3d8978c5 P 2d94505a8b6a4daab8fdbdffdf354596: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 12:46:16.753405 2726 log.cc:1196] T 94419de0777d47ea86a250fc3d8978c5 P 2d94505a8b6a4daab8fdbdffdf354596: Deleting WAL directory at /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/ts-2/wal/wals/94419de0777d47ea86a250fc3d8978c5
I20260927 12:46:16.753413 2727 ts_tablet_manager.cc:1939] T 94419de0777d47ea86a250fc3d8978c5 P c60eee98017b49d4aa748b9ac4846cde: Deleting consensus metadata
I20260927 12:46:16.753644 2726 ts_tablet_manager.cc:1939] T 94419de0777d47ea86a250fc3d8978c5 P 2d94505a8b6a4daab8fdbdffdf354596: Deleting consensus metadata
I20260927 12:46:16.754266 2126 catalog_manager.cc:4841] TS c60eee98017b49d4aa748b9ac4846cde (127.0.98.1:37687): tablet 94419de0777d47ea86a250fc3d8978c5 (table test-table [id=d2582fff44fc44359d95cc02f657a1ba]) successfully deleted
I20260927 12:46:16.754279 2124 catalog_manager.cc:4841] TS 2d94505a8b6a4daab8fdbdffdf354596 (127.0.98.3:45131): tablet 94419de0777d47ea86a250fc3d8978c5 (table test-table [id=d2582fff44fc44359d95cc02f657a1ba]) successfully deleted
I20260927 12:46:16.754534 2725 ts_tablet_manager.cc:1918] T 94419de0777d47ea86a250fc3d8978c5 P 1f0060ce040449f1a4415d377932fa44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 12:46:16.754624 2725 log.cc:1196] T 94419de0777d47ea86a250fc3d8978c5 P 1f0060ce040449f1a4415d377932fa44: Deleting WAL directory at /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/ts-1/wal/wals/94419de0777d47ea86a250fc3d8978c5
I20260927 12:46:16.754791 2725 ts_tablet_manager.cc:1939] T 94419de0777d47ea86a250fc3d8978c5 P 1f0060ce040449f1a4415d377932fa44: Deleting consensus metadata
I20260927 12:46:16.756229 2127 catalog_manager.cc:4841] TS 1f0060ce040449f1a4415d377932fa44 (127.0.98.2:45249): tablet 94419de0777d47ea86a250fc3d8978c5 (table test-table [id=d2582fff44fc44359d95cc02f657a1ba]) successfully deleted
Sep 27 12:46:16 dist-test-slave-lkn4 krb5kdc[2095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513176, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 12:46:16 dist-test-slave-lkn4 krb5kdc[2095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513176, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:46:16.777422 2734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:16.768802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55246 (local address 127.0.98.62:41189)
0927 12:46:16.769050 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:16.769057 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:16.769219 (+ 162us) server_negotiation.cc:407] Connection header received
0927 12:46:16.769335 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:16.769337 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:16.769380 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:16.769460 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:16.770176 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:16.770728 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:16.771594 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:16.771843 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:16.771932 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:16.774340 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:16.774343 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:16.774345 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:16.774372 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:16.775848 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:16.776261 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:16.776266 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:16.776267 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:16.776347 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:16.776588 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:16.776592 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:16.776593 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:16.776782 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:16.776905 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:16.777130 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 12:46:16.777252 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":97,"threads_started":1}
I20260927 12:46:16.780277 2139 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55246:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:46:16.780678 2139 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:46:16.785706 2557 tablet_service.cc:1467] Processing CreateTablet for tablet e0f87869bd3549d4ab02cc4661b9705b (DEFAULT_TABLE table=test-table [id=310f480ea3bd4fbbbaff3896cf720813]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:46:16.785702 2277 tablet_service.cc:1467] Processing CreateTablet for tablet e0f87869bd3549d4ab02cc4661b9705b (DEFAULT_TABLE table=test-table [id=310f480ea3bd4fbbbaff3896cf720813]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:46:16.785773 2417 tablet_service.cc:1467] Processing CreateTablet for tablet e0f87869bd3549d4ab02cc4661b9705b (DEFAULT_TABLE table=test-table [id=310f480ea3bd4fbbbaff3896cf720813]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:46:16.786010 2417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0f87869bd3549d4ab02cc4661b9705b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:46:16.786010 2277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0f87869bd3549d4ab02cc4661b9705b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:46:16.786011 2557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0f87869bd3549d4ab02cc4661b9705b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:46:16.788177 2736 tablet_bootstrap.cc:492] T e0f87869bd3549d4ab02cc4661b9705b P 2d94505a8b6a4daab8fdbdffdf354596: Bootstrap starting.
I20260927 12:46:16.788841 2737 tablet_bootstrap.cc:492] T e0f87869bd3549d4ab02cc4661b9705b P 1f0060ce040449f1a4415d377932fa44: Bootstrap starting.
I20260927 12:46:16.789029 2738 tablet_bootstrap.cc:492] T e0f87869bd3549d4ab02cc4661b9705b P c60eee98017b49d4aa748b9ac4846cde: Bootstrap starting.
I20260927 12:46:16.789727 2737 tablet_bootstrap.cc:654] T e0f87869bd3549d4ab02cc4661b9705b P 1f0060ce040449f1a4415d377932fa44: Neither blocks nor log segments found. Creating new log.
I20260927 12:46:16.789906 2738 tablet_bootstrap.cc:654] T e0f87869bd3549d4ab02cc4661b9705b P c60eee98017b49d4aa748b9ac4846cde: Neither blocks nor log segments found. Creating new log.
I20260927 12:46:16.790266 2736 tablet_bootstrap.cc:654] T e0f87869bd3549d4ab02cc4661b9705b P 2d94505a8b6a4daab8fdbdffdf354596: Neither blocks nor log segments found. Creating new log.
I20260927 12:46:16.790848 2737 tablet_bootstrap.cc:492] T e0f87869bd3549d4ab02cc4661b9705b P 1f0060ce040449f1a4415d377932fa44: No bootstrap required, opened a new log
I20260927 12:46:16.790937 2737 ts_tablet_manager.cc:1397] T e0f87869bd3549d4ab02cc4661b9705b P 1f0060ce040449f1a4415d377932fa44: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 12:46:16.791378 2737 raft_consensus.cc:348] T e0f87869bd3549d4ab02cc4661b9705b P 1f0060ce040449f1a4415d377932fa44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60eee98017b49d4aa748b9ac4846cde" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37687 } } peers { permanent_uuid: "1f0060ce040449f1a4415d377932fa44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45249 } } peers { permanent_uuid: "2d94505a8b6a4daab8fdbdffdf354596" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45131 } }
I20260927 12:46:16.791501 2737 raft_consensus.cc:374] T e0f87869bd3549d4ab02cc4661b9705b P 1f0060ce040449f1a4415d377932fa44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:46:16.791540 2737 raft_consensus.cc:729] T e0f87869bd3549d4ab02cc4661b9705b P 1f0060ce040449f1a4415d377932fa44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0060ce040449f1a4415d377932fa44, State: Initialized, Role: FOLLOWER
I20260927 12:46:16.791666 2737 consensus_queue.cc:260] T e0f87869bd3549d4ab02cc4661b9705b P 1f0060ce040449f1a4415d377932fa44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60eee98017b49d4aa748b9ac4846cde" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37687 } } peers { permanent_uuid: "1f0060ce040449f1a4415d377932fa44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45249 } } peers { permanent_uuid: "2d94505a8b6a4daab8fdbdffdf354596" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45131 } }
I20260927 12:46:16.792049 2737 ts_tablet_manager.cc:1428] T e0f87869bd3549d4ab02cc4661b9705b P 1f0060ce040449f1a4415d377932fa44: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 12:46:16.792318 2736 tablet_bootstrap.cc:492] T e0f87869bd3549d4ab02cc4661b9705b P 2d94505a8b6a4daab8fdbdffdf354596: No bootstrap required, opened a new log
I20260927 12:46:16.792460 2736 ts_tablet_manager.cc:1397] T e0f87869bd3549d4ab02cc4661b9705b P 2d94505a8b6a4daab8fdbdffdf354596: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 12:46:16.792922 2738 tablet_bootstrap.cc:492] T e0f87869bd3549d4ab02cc4661b9705b P c60eee98017b49d4aa748b9ac4846cde: No bootstrap required, opened a new log
I20260927 12:46:16.793017 2738 ts_tablet_manager.cc:1397] T e0f87869bd3549d4ab02cc4661b9705b P c60eee98017b49d4aa748b9ac4846cde: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260927 12:46:16.793455 2738 raft_consensus.cc:348] T e0f87869bd3549d4ab02cc4661b9705b P c60eee98017b49d4aa748b9ac4846cde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60eee98017b49d4aa748b9ac4846cde" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37687 } } peers { permanent_uuid: "1f0060ce040449f1a4415d377932fa44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45249 } } peers { permanent_uuid: "2d94505a8b6a4daab8fdbdffdf354596" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45131 } }
I20260927 12:46:16.793566 2738 raft_consensus.cc:374] T e0f87869bd3549d4ab02cc4661b9705b P c60eee98017b49d4aa748b9ac4846cde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:46:16.793602 2738 raft_consensus.cc:729] T e0f87869bd3549d4ab02cc4661b9705b P c60eee98017b49d4aa748b9ac4846cde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c60eee98017b49d4aa748b9ac4846cde, State: Initialized, Role: FOLLOWER
I20260927 12:46:16.793646 2736 raft_consensus.cc:348] T e0f87869bd3549d4ab02cc4661b9705b P 2d94505a8b6a4daab8fdbdffdf354596 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60eee98017b49d4aa748b9ac4846cde" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37687 } } peers { permanent_uuid: "1f0060ce040449f1a4415d377932fa44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45249 } } peers { permanent_uuid: "2d94505a8b6a4daab8fdbdffdf354596" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45131 } }
I20260927 12:46:16.793809 2736 raft_consensus.cc:374] T e0f87869bd3549d4ab02cc4661b9705b P 2d94505a8b6a4daab8fdbdffdf354596 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:46:16.793859 2736 raft_consensus.cc:729] T e0f87869bd3549d4ab02cc4661b9705b P 2d94505a8b6a4daab8fdbdffdf354596 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d94505a8b6a4daab8fdbdffdf354596, State: Initialized, Role: FOLLOWER
I20260927 12:46:16.793846 2738 consensus_queue.cc:260] T e0f87869bd3549d4ab02cc4661b9705b P c60eee98017b49d4aa748b9ac4846cde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60eee98017b49d4aa748b9ac4846cde" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37687 } } peers { permanent_uuid: "1f0060ce040449f1a4415d377932fa44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45249 } } peers { permanent_uuid: "2d94505a8b6a4daab8fdbdffdf354596" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45131 } }
I20260927 12:46:16.794013 2736 consensus_queue.cc:260] T e0f87869bd3549d4ab02cc4661b9705b P 2d94505a8b6a4daab8fdbdffdf354596 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60eee98017b49d4aa748b9ac4846cde" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37687 } } peers { permanent_uuid: "1f0060ce040449f1a4415d377932fa44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45249 } } peers { permanent_uuid: "2d94505a8b6a4daab8fdbdffdf354596" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45131 } }
I20260927 12:46:16.794255 2738 ts_tablet_manager.cc:1428] T e0f87869bd3549d4ab02cc4661b9705b P c60eee98017b49d4aa748b9ac4846cde: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260927 12:46:16.794749 2736 ts_tablet_manager.cc:1428] T e0f87869bd3549d4ab02cc4661b9705b P 2d94505a8b6a4daab8fdbdffdf354596: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260927 12:46:17.007145 2742 raft_consensus.cc:482] T e0f87869bd3549d4ab02cc4661b9705b P 1f0060ce040449f1a4415d377932fa44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:46:17.007325 2742 raft_consensus.cc:504] T e0f87869bd3549d4ab02cc4661b9705b P 1f0060ce040449f1a4415d377932fa44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60eee98017b49d4aa748b9ac4846cde" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37687 } } peers { permanent_uuid: "1f0060ce040449f1a4415d377932fa44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45249 } } peers { permanent_uuid: "2d94505a8b6a4daab8fdbdffdf354596" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45131 } }
I20260927 12:46:17.007776 2742 leader_election.cc:288] T e0f87869bd3549d4ab02cc4661b9705b P 1f0060ce040449f1a4415d377932fa44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c60eee98017b49d4aa748b9ac4846cde (127.0.98.1:37687), 2d94505a8b6a4daab8fdbdffdf354596 (127.0.98.3:45131)
I20260927 12:46:17.008255 2297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f87869bd3549d4ab02cc4661b9705b" candidate_uuid: "1f0060ce040449f1a4415d377932fa44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c60eee98017b49d4aa748b9ac4846cde" is_pre_election: true
I20260927 12:46:17.008251 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f87869bd3549d4ab02cc4661b9705b" candidate_uuid: "1f0060ce040449f1a4415d377932fa44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d94505a8b6a4daab8fdbdffdf354596" is_pre_election: true
I20260927 12:46:17.008422 2297 raft_consensus.cc:2454] T e0f87869bd3549d4ab02cc4661b9705b P c60eee98017b49d4aa748b9ac4846cde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f0060ce040449f1a4415d377932fa44 in term 0.
I20260927 12:46:17.008440 2577 raft_consensus.cc:2454] T e0f87869bd3549d4ab02cc4661b9705b P 2d94505a8b6a4daab8fdbdffdf354596 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f0060ce040449f1a4415d377932fa44 in term 0.
I20260927 12:46:17.008791 2368 leader_election.cc:302] T e0f87869bd3549d4ab02cc4661b9705b P 1f0060ce040449f1a4415d377932fa44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0060ce040449f1a4415d377932fa44, 2d94505a8b6a4daab8fdbdffdf354596; no voters:
I20260927 12:46:17.008996 2742 raft_consensus.cc:2781] T e0f87869bd3549d4ab02cc4661b9705b P 1f0060ce040449f1a4415d377932fa44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:46:17.009063 2742 raft_consensus.cc:482] T e0f87869bd3549d4ab02cc4661b9705b P 1f0060ce040449f1a4415d377932fa44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:46:17.009090 2742 raft_consensus.cc:3037] T e0f87869bd3549d4ab02cc4661b9705b P 1f0060ce040449f1a4415d377932fa44 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:46:17.010167 2742 raft_consensus.cc:504] T e0f87869bd3549d4ab02cc4661b9705b P 1f0060ce040449f1a4415d377932fa44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60eee98017b49d4aa748b9ac4846cde" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37687 } } peers { permanent_uuid: "1f0060ce040449f1a4415d377932fa44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45249 } } peers { permanent_uuid: "2d94505a8b6a4daab8fdbdffdf354596" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45131 } }
I20260927 12:46:17.010509 2742 leader_election.cc:288] T e0f87869bd3549d4ab02cc4661b9705b P 1f0060ce040449f1a4415d377932fa44 [CANDIDATE]: Term 1 election: Requested vote from peers c60eee98017b49d4aa748b9ac4846cde (127.0.98.1:37687), 2d94505a8b6a4daab8fdbdffdf354596 (127.0.98.3:45131)
I20260927 12:46:17.010903 2297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f87869bd3549d4ab02cc4661b9705b" candidate_uuid: "1f0060ce040449f1a4415d377932fa44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c60eee98017b49d4aa748b9ac4846cde"
I20260927 12:46:17.010903 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f87869bd3549d4ab02cc4661b9705b" candidate_uuid: "1f0060ce040449f1a4415d377932fa44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d94505a8b6a4daab8fdbdffdf354596"
I20260927 12:46:17.011021 2297 raft_consensus.cc:3037] T e0f87869bd3549d4ab02cc4661b9705b P c60eee98017b49d4aa748b9ac4846cde [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:46:17.011027 2577 raft_consensus.cc:3037] T e0f87869bd3549d4ab02cc4661b9705b P 2d94505a8b6a4daab8fdbdffdf354596 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:46:17.011896 2297 raft_consensus.cc:2454] T e0f87869bd3549d4ab02cc4661b9705b P c60eee98017b49d4aa748b9ac4846cde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f0060ce040449f1a4415d377932fa44 in term 1.
I20260927 12:46:17.011972 2577 raft_consensus.cc:2454] T e0f87869bd3549d4ab02cc4661b9705b P 2d94505a8b6a4daab8fdbdffdf354596 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f0060ce040449f1a4415d377932fa44 in term 1.
I20260927 12:46:17.012203 2370 leader_election.cc:302] T e0f87869bd3549d4ab02cc4661b9705b P 1f0060ce040449f1a4415d377932fa44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0060ce040449f1a4415d377932fa44, c60eee98017b49d4aa748b9ac4846cde; no voters:
I20260927 12:46:17.012399 2742 raft_consensus.cc:2781] T e0f87869bd3549d4ab02cc4661b9705b P 1f0060ce040449f1a4415d377932fa44 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:46:17.012619 2742 raft_consensus.cc:686] T e0f87869bd3549d4ab02cc4661b9705b P 1f0060ce040449f1a4415d377932fa44 [term 1 LEADER]: Becoming Leader. State: Replica: 1f0060ce040449f1a4415d377932fa44, State: Running, Role: LEADER
I20260927 12:46:17.012768 2742 consensus_queue.cc:237] T e0f87869bd3549d4ab02cc4661b9705b P 1f0060ce040449f1a4415d377932fa44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60eee98017b49d4aa748b9ac4846cde" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37687 } } peers { permanent_uuid: "1f0060ce040449f1a4415d377932fa44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45249 } } peers { permanent_uuid: "2d94505a8b6a4daab8fdbdffdf354596" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45131 } }
I20260927 12:46:17.014464 2139 catalog_manager.cc:5495] T e0f87869bd3549d4ab02cc4661b9705b P 1f0060ce040449f1a4415d377932fa44 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f0060ce040449f1a4415d377932fa44 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "1f0060ce040449f1a4415d377932fa44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60eee98017b49d4aa748b9ac4846cde" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f0060ce040449f1a4415d377932fa44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45249 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d94505a8b6a4daab8fdbdffdf354596" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45131 } health_report { overall_health: UNKNOWN } } }
I20260927 12:46:17.037813 2747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:17.034321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36402 (local address 127.0.98.2:45249)
0927 12:46:17.034607 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:17.034613 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:17.034625 (+ 12us) server_negotiation.cc:407] Connection header received
0927 12:46:17.034665 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:17.034667 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:17.034709 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:17.034780 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:46:17.035088 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:17.035659 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:17.036570 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:17.036825 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:17.036911 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:17.037291 (+ 380us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:46:17.037381 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:46:17.037441 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:17.037521 (+ 80us) server_negotiation.cc:299] Negotiation successful
0927 12:46:17.037592 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":136,"threads_started":1}
I20260927 12:46:17.044577 2577 raft_consensus.cc:1261] T e0f87869bd3549d4ab02cc4661b9705b P 2d94505a8b6a4daab8fdbdffdf354596 [term 1 FOLLOWER]: Refusing update from remote peer 1f0060ce040449f1a4415d377932fa44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:46:17.044622 2297 raft_consensus.cc:1261] T e0f87869bd3549d4ab02cc4661b9705b P c60eee98017b49d4aa748b9ac4846cde [term 1 FOLLOWER]: Refusing update from remote peer 1f0060ce040449f1a4415d377932fa44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:46:17.044961 2744 consensus_queue.cc:1035] T e0f87869bd3549d4ab02cc4661b9705b P 1f0060ce040449f1a4415d377932fa44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d94505a8b6a4daab8fdbdffdf354596" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:46:17.045121 2742 consensus_queue.cc:1035] T e0f87869bd3549d4ab02cc4661b9705b P 1f0060ce040449f1a4415d377932fa44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c60eee98017b49d4aa748b9ac4846cde" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:46:17.074340 2137 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:46:17.074520 2137 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:46:17.076514 2137 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 92ba4aec5bf34fde9f6049baedcb6f3f: Sending DeleteTablet for 3 replicas of tablet e0f87869bd3549d4ab02cc4661b9705b
I20260927 12:46:17.077066 2417 tablet_service.cc:1514] Processing DeleteTablet for tablet e0f87869bd3549d4ab02cc4661b9705b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:46:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36378
I20260927 12:46:17.077172 2277 tablet_service.cc:1514] Processing DeleteTablet for tablet e0f87869bd3549d4ab02cc4661b9705b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:46:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39256
I20260927 12:46:17.077173 2557 tablet_service.cc:1514] Processing DeleteTablet for tablet e0f87869bd3549d4ab02cc4661b9705b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:46:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48880
I20260927 12:46:17.077266 2725 tablet_replica.cc:330] T e0f87869bd3549d4ab02cc4661b9705b P 1f0060ce040449f1a4415d377932fa44: stopping tablet replica
I20260927 12:46:17.077337 2727 tablet_replica.cc:330] T e0f87869bd3549d4ab02cc4661b9705b P c60eee98017b49d4aa748b9ac4846cde: stopping tablet replica
I20260927 12:46:17.077358 2726 tablet_replica.cc:330] T e0f87869bd3549d4ab02cc4661b9705b P 2d94505a8b6a4daab8fdbdffdf354596: stopping tablet replica
I20260927 12:46:17.077384 2725 raft_consensus.cc:2229] T e0f87869bd3549d4ab02cc4661b9705b P 1f0060ce040449f1a4415d377932fa44 [term 1 LEADER]: Raft consensus shutting down.
I20260927 12:46:17.077430 2727 raft_consensus.cc:2229] T e0f87869bd3549d4ab02cc4661b9705b P c60eee98017b49d4aa748b9ac4846cde [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:46:17.077458 2726 raft_consensus.cc:2229] T e0f87869bd3549d4ab02cc4661b9705b P 2d94505a8b6a4daab8fdbdffdf354596 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:46:17.077530 2727 raft_consensus.cc:2258] T e0f87869bd3549d4ab02cc4661b9705b P c60eee98017b49d4aa748b9ac4846cde [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:46:17.077560 2726 raft_consensus.cc:2258] T e0f87869bd3549d4ab02cc4661b9705b P 2d94505a8b6a4daab8fdbdffdf354596 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:46:17.077558 2725 raft_consensus.cc:2258] T e0f87869bd3549d4ab02cc4661b9705b P 1f0060ce040449f1a4415d377932fa44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:46:17.078234 2725 ts_tablet_manager.cc:1905] T e0f87869bd3549d4ab02cc4661b9705b P 1f0060ce040449f1a4415d377932fa44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:46:17.078234 2727 ts_tablet_manager.cc:1905] T e0f87869bd3549d4ab02cc4661b9705b P c60eee98017b49d4aa748b9ac4846cde: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:46:17.078497 2137 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55246:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:46:17.078892 2137 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:46:17.080744 2725 ts_tablet_manager.cc:1918] T e0f87869bd3549d4ab02cc4661b9705b P 1f0060ce040449f1a4415d377932fa44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 12:46:17.080744 2727 ts_tablet_manager.cc:1918] T e0f87869bd3549d4ab02cc4661b9705b P c60eee98017b49d4aa748b9ac4846cde: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 12:46:17.080822 2725 log.cc:1196] T e0f87869bd3549d4ab02cc4661b9705b P 1f0060ce040449f1a4415d377932fa44: Deleting WAL directory at /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/ts-1/wal/wals/e0f87869bd3549d4ab02cc4661b9705b
I20260927 12:46:17.080828 2727 log.cc:1196] T e0f87869bd3549d4ab02cc4661b9705b P c60eee98017b49d4aa748b9ac4846cde: Deleting WAL directory at /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/ts-0/wal/wals/e0f87869bd3549d4ab02cc4661b9705b
I20260927 12:46:17.081055 2725 ts_tablet_manager.cc:1939] T e0f87869bd3549d4ab02cc4661b9705b P 1f0060ce040449f1a4415d377932fa44: Deleting consensus metadata
I20260927 12:46:17.081055 2727 ts_tablet_manager.cc:1939] T e0f87869bd3549d4ab02cc4661b9705b P c60eee98017b49d4aa748b9ac4846cde: Deleting consensus metadata
I20260927 12:46:17.081722 2726 ts_tablet_manager.cc:1905] T e0f87869bd3549d4ab02cc4661b9705b P 2d94505a8b6a4daab8fdbdffdf354596: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:46:17.081837 2126 catalog_manager.cc:4841] TS c60eee98017b49d4aa748b9ac4846cde (127.0.98.1:37687): tablet e0f87869bd3549d4ab02cc4661b9705b (table test-table [id=310f480ea3bd4fbbbaff3896cf720813]) successfully deleted
I20260927 12:46:17.081920 2127 catalog_manager.cc:4841] TS 1f0060ce040449f1a4415d377932fa44 (127.0.98.2:45249): tablet e0f87869bd3549d4ab02cc4661b9705b (table test-table [id=310f480ea3bd4fbbbaff3896cf720813]) successfully deleted
I20260927 12:46:17.083367 2417 tablet_service.cc:1467] Processing CreateTablet for tablet 8d0cedcd884b48b996538fe78912e181 (DEFAULT_TABLE table=test-table [id=6f4999bc5f634b3c8811c328197b9cea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:46:17.083374 2277 tablet_service.cc:1467] Processing CreateTablet for tablet 8d0cedcd884b48b996538fe78912e181 (DEFAULT_TABLE table=test-table [id=6f4999bc5f634b3c8811c328197b9cea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:46:17.083606 2417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d0cedcd884b48b996538fe78912e181. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:46:17.083612 2277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d0cedcd884b48b996538fe78912e181. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:46:17.084156 2726 ts_tablet_manager.cc:1918] T e0f87869bd3549d4ab02cc4661b9705b P 2d94505a8b6a4daab8fdbdffdf354596: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 12:46:17.084227 2726 log.cc:1196] T e0f87869bd3549d4ab02cc4661b9705b P 2d94505a8b6a4daab8fdbdffdf354596: Deleting WAL directory at /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/ts-2/wal/wals/e0f87869bd3549d4ab02cc4661b9705b
I20260927 12:46:17.084460 2726 ts_tablet_manager.cc:1939] T e0f87869bd3549d4ab02cc4661b9705b P 2d94505a8b6a4daab8fdbdffdf354596: Deleting consensus metadata
I20260927 12:46:17.085172 2124 catalog_manager.cc:4841] TS 2d94505a8b6a4daab8fdbdffdf354596 (127.0.98.3:45131): tablet e0f87869bd3549d4ab02cc4661b9705b (table test-table [id=310f480ea3bd4fbbbaff3896cf720813]) successfully deleted
I20260927 12:46:17.085507 2738 tablet_bootstrap.cc:492] T 8d0cedcd884b48b996538fe78912e181 P c60eee98017b49d4aa748b9ac4846cde: Bootstrap starting.
I20260927 12:46:17.085878 2557 tablet_service.cc:1467] Processing CreateTablet for tablet 8d0cedcd884b48b996538fe78912e181 (DEFAULT_TABLE table=test-table [id=6f4999bc5f634b3c8811c328197b9cea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:46:17.086128 2557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d0cedcd884b48b996538fe78912e181. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:46:17.086407 2738 tablet_bootstrap.cc:654] T 8d0cedcd884b48b996538fe78912e181 P c60eee98017b49d4aa748b9ac4846cde: Neither blocks nor log segments found. Creating new log.
I20260927 12:46:17.087495 2738 tablet_bootstrap.cc:492] T 8d0cedcd884b48b996538fe78912e181 P c60eee98017b49d4aa748b9ac4846cde: No bootstrap required, opened a new log
I20260927 12:46:17.087579 2738 ts_tablet_manager.cc:1397] T 8d0cedcd884b48b996538fe78912e181 P c60eee98017b49d4aa748b9ac4846cde: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 12:46:17.088006 2738 raft_consensus.cc:348] T 8d0cedcd884b48b996538fe78912e181 P c60eee98017b49d4aa748b9ac4846cde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60eee98017b49d4aa748b9ac4846cde" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37687 } } peers { permanent_uuid: "1f0060ce040449f1a4415d377932fa44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45249 } } peers { permanent_uuid: "2d94505a8b6a4daab8fdbdffdf354596" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45131 } }
I20260927 12:46:17.088120 2738 raft_consensus.cc:374] T 8d0cedcd884b48b996538fe78912e181 P c60eee98017b49d4aa748b9ac4846cde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:46:17.088157 2738 raft_consensus.cc:729] T 8d0cedcd884b48b996538fe78912e181 P c60eee98017b49d4aa748b9ac4846cde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c60eee98017b49d4aa748b9ac4846cde, State: Initialized, Role: FOLLOWER
I20260927 12:46:17.088286 2738 consensus_queue.cc:260] T 8d0cedcd884b48b996538fe78912e181 P c60eee98017b49d4aa748b9ac4846cde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60eee98017b49d4aa748b9ac4846cde" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37687 } } peers { permanent_uuid: "1f0060ce040449f1a4415d377932fa44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45249 } } peers { permanent_uuid: "2d94505a8b6a4daab8fdbdffdf354596" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45131 } }
I20260927 12:46:17.088547 2738 ts_tablet_manager.cc:1428] T 8d0cedcd884b48b996538fe78912e181 P c60eee98017b49d4aa748b9ac4846cde: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260927 12:46:17.089214 2736 tablet_bootstrap.cc:492] T 8d0cedcd884b48b996538fe78912e181 P 2d94505a8b6a4daab8fdbdffdf354596: Bootstrap starting.
I20260927 12:46:17.089388 2737 tablet_bootstrap.cc:492] T 8d0cedcd884b48b996538fe78912e181 P 1f0060ce040449f1a4415d377932fa44: Bootstrap starting.
I20260927 12:46:17.089958 2736 tablet_bootstrap.cc:654] T 8d0cedcd884b48b996538fe78912e181 P 2d94505a8b6a4daab8fdbdffdf354596: Neither blocks nor log segments found. Creating new log.
I20260927 12:46:17.090094 2737 tablet_bootstrap.cc:654] T 8d0cedcd884b48b996538fe78912e181 P 1f0060ce040449f1a4415d377932fa44: Neither blocks nor log segments found. Creating new log.
I20260927 12:46:17.090934 2736 tablet_bootstrap.cc:492] T 8d0cedcd884b48b996538fe78912e181 P 2d94505a8b6a4daab8fdbdffdf354596: No bootstrap required, opened a new log
I20260927 12:46:17.091020 2736 ts_tablet_manager.cc:1397] T 8d0cedcd884b48b996538fe78912e181 P 2d94505a8b6a4daab8fdbdffdf354596: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 12:46:17.091146 2737 tablet_bootstrap.cc:492] T 8d0cedcd884b48b996538fe78912e181 P 1f0060ce040449f1a4415d377932fa44: No bootstrap required, opened a new log
I20260927 12:46:17.091223 2737 ts_tablet_manager.cc:1397] T 8d0cedcd884b48b996538fe78912e181 P 1f0060ce040449f1a4415d377932fa44: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 12:46:17.091404 2736 raft_consensus.cc:348] T 8d0cedcd884b48b996538fe78912e181 P 2d94505a8b6a4daab8fdbdffdf354596 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60eee98017b49d4aa748b9ac4846cde" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37687 } } peers { permanent_uuid: "1f0060ce040449f1a4415d377932fa44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45249 } } peers { permanent_uuid: "2d94505a8b6a4daab8fdbdffdf354596" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45131 } }
I20260927 12:46:17.091516 2736 raft_consensus.cc:374] T 8d0cedcd884b48b996538fe78912e181 P 2d94505a8b6a4daab8fdbdffdf354596 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:46:17.091554 2736 raft_consensus.cc:729] T 8d0cedcd884b48b996538fe78912e181 P 2d94505a8b6a4daab8fdbdffdf354596 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d94505a8b6a4daab8fdbdffdf354596, State: Initialized, Role: FOLLOWER
I20260927 12:46:17.091566 2737 raft_consensus.cc:348] T 8d0cedcd884b48b996538fe78912e181 P 1f0060ce040449f1a4415d377932fa44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60eee98017b49d4aa748b9ac4846cde" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37687 } } peers { permanent_uuid: "1f0060ce040449f1a4415d377932fa44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45249 } } peers { permanent_uuid: "2d94505a8b6a4daab8fdbdffdf354596" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45131 } }
I20260927 12:46:17.091662 2737 raft_consensus.cc:374] T 8d0cedcd884b48b996538fe78912e181 P 1f0060ce040449f1a4415d377932fa44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:46:17.091696 2737 raft_consensus.cc:729] T 8d0cedcd884b48b996538fe78912e181 P 1f0060ce040449f1a4415d377932fa44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f0060ce040449f1a4415d377932fa44, State: Initialized, Role: FOLLOWER
I20260927 12:46:17.091691 2736 consensus_queue.cc:260] T 8d0cedcd884b48b996538fe78912e181 P 2d94505a8b6a4daab8fdbdffdf354596 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60eee98017b49d4aa748b9ac4846cde" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37687 } } peers { permanent_uuid: "1f0060ce040449f1a4415d377932fa44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45249 } } peers { permanent_uuid: "2d94505a8b6a4daab8fdbdffdf354596" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45131 } }
I20260927 12:46:17.091796 2737 consensus_queue.cc:260] T 8d0cedcd884b48b996538fe78912e181 P 1f0060ce040449f1a4415d377932fa44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60eee98017b49d4aa748b9ac4846cde" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37687 } } peers { permanent_uuid: "1f0060ce040449f1a4415d377932fa44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45249 } } peers { permanent_uuid: "2d94505a8b6a4daab8fdbdffdf354596" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45131 } }
I20260927 12:46:17.092041 2736 ts_tablet_manager.cc:1428] T 8d0cedcd884b48b996538fe78912e181 P 2d94505a8b6a4daab8fdbdffdf354596: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 12:46:17.092053 2737 ts_tablet_manager.cc:1428] T 8d0cedcd884b48b996538fe78912e181 P 1f0060ce040449f1a4415d377932fa44: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 12:46:17.436408 2744 raft_consensus.cc:482] T 8d0cedcd884b48b996538fe78912e181 P 1f0060ce040449f1a4415d377932fa44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:46:17.436583 2744 raft_consensus.cc:504] T 8d0cedcd884b48b996538fe78912e181 P 1f0060ce040449f1a4415d377932fa44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60eee98017b49d4aa748b9ac4846cde" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37687 } } peers { permanent_uuid: "1f0060ce040449f1a4415d377932fa44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45249 } } peers { permanent_uuid: "2d94505a8b6a4daab8fdbdffdf354596" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45131 } }
I20260927 12:46:17.437050 2744 leader_election.cc:288] T 8d0cedcd884b48b996538fe78912e181 P 1f0060ce040449f1a4415d377932fa44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c60eee98017b49d4aa748b9ac4846cde (127.0.98.1:37687), 2d94505a8b6a4daab8fdbdffdf354596 (127.0.98.3:45131)
I20260927 12:46:17.437542 2296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d0cedcd884b48b996538fe78912e181" candidate_uuid: "1f0060ce040449f1a4415d377932fa44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c60eee98017b49d4aa748b9ac4846cde" is_pre_election: true
I20260927 12:46:17.437558 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d0cedcd884b48b996538fe78912e181" candidate_uuid: "1f0060ce040449f1a4415d377932fa44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d94505a8b6a4daab8fdbdffdf354596" is_pre_election: true
I20260927 12:46:17.437690 2296 raft_consensus.cc:2454] T 8d0cedcd884b48b996538fe78912e181 P c60eee98017b49d4aa748b9ac4846cde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f0060ce040449f1a4415d377932fa44 in term 0.
I20260927 12:46:17.437688 2577 raft_consensus.cc:2454] T 8d0cedcd884b48b996538fe78912e181 P 2d94505a8b6a4daab8fdbdffdf354596 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f0060ce040449f1a4415d377932fa44 in term 0.
I20260927 12:46:17.438023 2370 leader_election.cc:302] T 8d0cedcd884b48b996538fe78912e181 P 1f0060ce040449f1a4415d377932fa44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0060ce040449f1a4415d377932fa44, c60eee98017b49d4aa748b9ac4846cde; no voters:
I20260927 12:46:17.438230 2744 raft_consensus.cc:2781] T 8d0cedcd884b48b996538fe78912e181 P 1f0060ce040449f1a4415d377932fa44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:46:17.438283 2744 raft_consensus.cc:482] T 8d0cedcd884b48b996538fe78912e181 P 1f0060ce040449f1a4415d377932fa44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:46:17.438310 2744 raft_consensus.cc:3037] T 8d0cedcd884b48b996538fe78912e181 P 1f0060ce040449f1a4415d377932fa44 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:46:17.439273 2744 raft_consensus.cc:504] T 8d0cedcd884b48b996538fe78912e181 P 1f0060ce040449f1a4415d377932fa44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60eee98017b49d4aa748b9ac4846cde" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37687 } } peers { permanent_uuid: "1f0060ce040449f1a4415d377932fa44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45249 } } peers { permanent_uuid: "2d94505a8b6a4daab8fdbdffdf354596" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45131 } }
I20260927 12:46:17.439604 2744 leader_election.cc:288] T 8d0cedcd884b48b996538fe78912e181 P 1f0060ce040449f1a4415d377932fa44 [CANDIDATE]: Term 1 election: Requested vote from peers c60eee98017b49d4aa748b9ac4846cde (127.0.98.1:37687), 2d94505a8b6a4daab8fdbdffdf354596 (127.0.98.3:45131)
I20260927 12:46:17.440016 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d0cedcd884b48b996538fe78912e181" candidate_uuid: "1f0060ce040449f1a4415d377932fa44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d94505a8b6a4daab8fdbdffdf354596"
I20260927 12:46:17.440138 2577 raft_consensus.cc:3037] T 8d0cedcd884b48b996538fe78912e181 P 2d94505a8b6a4daab8fdbdffdf354596 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:46:17.440131 2296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d0cedcd884b48b996538fe78912e181" candidate_uuid: "1f0060ce040449f1a4415d377932fa44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c60eee98017b49d4aa748b9ac4846cde"
I20260927 12:46:17.440263 2296 raft_consensus.cc:3037] T 8d0cedcd884b48b996538fe78912e181 P c60eee98017b49d4aa748b9ac4846cde [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:46:17.441085 2577 raft_consensus.cc:2454] T 8d0cedcd884b48b996538fe78912e181 P 2d94505a8b6a4daab8fdbdffdf354596 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f0060ce040449f1a4415d377932fa44 in term 1.
I20260927 12:46:17.441172 2296 raft_consensus.cc:2454] T 8d0cedcd884b48b996538fe78912e181 P c60eee98017b49d4aa748b9ac4846cde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f0060ce040449f1a4415d377932fa44 in term 1.
I20260927 12:46:17.441356 2368 leader_election.cc:302] T 8d0cedcd884b48b996538fe78912e181 P 1f0060ce040449f1a4415d377932fa44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f0060ce040449f1a4415d377932fa44, 2d94505a8b6a4daab8fdbdffdf354596; no voters:
I20260927 12:46:17.441536 2744 raft_consensus.cc:2781] T 8d0cedcd884b48b996538fe78912e181 P 1f0060ce040449f1a4415d377932fa44 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:46:17.441639 2744 raft_consensus.cc:686] T 8d0cedcd884b48b996538fe78912e181 P 1f0060ce040449f1a4415d377932fa44 [term 1 LEADER]: Becoming Leader. State: Replica: 1f0060ce040449f1a4415d377932fa44, State: Running, Role: LEADER
I20260927 12:46:17.441778 2744 consensus_queue.cc:237] T 8d0cedcd884b48b996538fe78912e181 P 1f0060ce040449f1a4415d377932fa44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60eee98017b49d4aa748b9ac4846cde" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37687 } } peers { permanent_uuid: "1f0060ce040449f1a4415d377932fa44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45249 } } peers { permanent_uuid: "2d94505a8b6a4daab8fdbdffdf354596" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45131 } }
I20260927 12:46:17.443439 2137 catalog_manager.cc:5495] T 8d0cedcd884b48b996538fe78912e181 P 1f0060ce040449f1a4415d377932fa44 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f0060ce040449f1a4415d377932fa44 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "1f0060ce040449f1a4415d377932fa44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c60eee98017b49d4aa748b9ac4846cde" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f0060ce040449f1a4415d377932fa44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45249 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d94505a8b6a4daab8fdbdffdf354596" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45131 } health_report { overall_health: UNKNOWN } } }
I20260927 12:46:17.466840 2577 raft_consensus.cc:1261] T 8d0cedcd884b48b996538fe78912e181 P 2d94505a8b6a4daab8fdbdffdf354596 [term 1 FOLLOWER]: Refusing update from remote peer 1f0060ce040449f1a4415d377932fa44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:46:17.466840 2296 raft_consensus.cc:1261] T 8d0cedcd884b48b996538fe78912e181 P c60eee98017b49d4aa748b9ac4846cde [term 1 FOLLOWER]: Refusing update from remote peer 1f0060ce040449f1a4415d377932fa44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:46:17.467209 2744 consensus_queue.cc:1035] T 8d0cedcd884b48b996538fe78912e181 P 1f0060ce040449f1a4415d377932fa44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d94505a8b6a4daab8fdbdffdf354596" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:46:17.467373 2742 consensus_queue.cc:1035] T 8d0cedcd884b48b996538fe78912e181 P 1f0060ce040449f1a4415d377932fa44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c60eee98017b49d4aa748b9ac4846cde" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 12:46:17.479812 2709 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 12:46:19.028012 2137 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:46:19.028163 2137 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:46:19.030524 2137 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 92ba4aec5bf34fde9f6049baedcb6f3f: Sending DeleteTablet for 3 replicas of tablet 8d0cedcd884b48b996538fe78912e181
I20260927 12:46:19.031066 2417 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d0cedcd884b48b996538fe78912e181 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:46:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36378
I20260927 12:46:19.031108 2277 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d0cedcd884b48b996538fe78912e181 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:46:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39256
I20260927 12:46:19.031409 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 2212
I20260927 12:46:19.031414 2557 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d0cedcd884b48b996538fe78912e181 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:46:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48880
I20260927 12:46:19.031474 2779 tablet_replica.cc:330] T 8d0cedcd884b48b996538fe78912e181 P 1f0060ce040449f1a4415d377932fa44: stopping tablet replica
I20260927 12:46:19.032728 2779 raft_consensus.cc:2229] T 8d0cedcd884b48b996538fe78912e181 P 1f0060ce040449f1a4415d377932fa44 [term 1 LEADER]: Raft consensus shutting down.
I20260927 12:46:19.032940 2779 raft_consensus.cc:2258] T 8d0cedcd884b48b996538fe78912e181 P 1f0060ce040449f1a4415d377932fa44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:46:19.033165 2781 tablet_replica.cc:330] T 8d0cedcd884b48b996538fe78912e181 P 2d94505a8b6a4daab8fdbdffdf354596: stopping tablet replica
I20260927 12:46:19.033351 2781 raft_consensus.cc:2229] T 8d0cedcd884b48b996538fe78912e181 P 2d94505a8b6a4daab8fdbdffdf354596 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:46:19.033468 2781 raft_consensus.cc:2258] T 8d0cedcd884b48b996538fe78912e181 P 2d94505a8b6a4daab8fdbdffdf354596 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:46:19.033721 2779 ts_tablet_manager.cc:1905] T 8d0cedcd884b48b996538fe78912e181 P 1f0060ce040449f1a4415d377932fa44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:46:19.034075 2781 ts_tablet_manager.cc:1905] T 8d0cedcd884b48b996538fe78912e181 P 2d94505a8b6a4daab8fdbdffdf354596: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:46:19.035862 2779 ts_tablet_manager.cc:1918] T 8d0cedcd884b48b996538fe78912e181 P 1f0060ce040449f1a4415d377932fa44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 12:46:19.035930 2779 log.cc:1196] T 8d0cedcd884b48b996538fe78912e181 P 1f0060ce040449f1a4415d377932fa44: Deleting WAL directory at /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/ts-1/wal/wals/8d0cedcd884b48b996538fe78912e181
I20260927 12:46:19.036164 2779 ts_tablet_manager.cc:1939] T 8d0cedcd884b48b996538fe78912e181 P 1f0060ce040449f1a4415d377932fa44: Deleting consensus metadata
I20260927 12:46:19.036528 2781 ts_tablet_manager.cc:1918] T 8d0cedcd884b48b996538fe78912e181 P 2d94505a8b6a4daab8fdbdffdf354596: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 12:46:19.036600 2781 log.cc:1196] T 8d0cedcd884b48b996538fe78912e181 P 2d94505a8b6a4daab8fdbdffdf354596: Deleting WAL directory at /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790513161160093-392-0/minicluster-data/ts-2/wal/wals/8d0cedcd884b48b996538fe78912e181
I20260927 12:46:19.036815 2781 ts_tablet_manager.cc:1939] T 8d0cedcd884b48b996538fe78912e181 P 2d94505a8b6a4daab8fdbdffdf354596: Deleting consensus metadata
I20260927 12:46:19.036787 2127 catalog_manager.cc:4841] TS 1f0060ce040449f1a4415d377932fa44 (127.0.98.2:45249): tablet 8d0cedcd884b48b996538fe78912e181 (table test-table [id=6f4999bc5f634b3c8811c328197b9cea]) successfully deleted
I20260927 12:46:19.037482 2124 catalog_manager.cc:4841] TS 2d94505a8b6a4daab8fdbdffdf354596 (127.0.98.3:45131): tablet 8d0cedcd884b48b996538fe78912e181 (table test-table [id=6f4999bc5f634b3c8811c328197b9cea]) successfully deleted
I20260927 12:46:19.038897 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 2352
W20260927 12:46:19.038911 2126 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37687 (error 108)
I20260927 12:46:19.040331 2783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:19.039947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37687 (local address 127.0.0.1:39268)
0927 12:46:19.040183 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:19.040247 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":83,"threads_started":1}
W20260927 12:46:19.040452 2126 catalog_manager.cc:4584] TS c60eee98017b49d4aa748b9ac4846cde (127.0.98.1:37687): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8d0cedcd884b48b996538fe78912e181: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37687: connect: Connection refused (error 111)
I20260927 12:46:19.045159 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 2492
I20260927 12:46:19.050755 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 2108
2026-09-27T12:46:19Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6515 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:46:19 dist-test-slave-lkn4 krb5kdc[2786](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:46:19 dist-test-slave-lkn4 krb5kdc[2786](info): set up 2 sockets
Sep 27 12:46:19 dist-test-slave-lkn4 krb5kdc[2786](info): commencing operation
krb5kdc: starting...
W20260927 12:46:20.017976 392 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/krb5kdc/test-user.keytab.
Sep 27 12:46:20 dist-test-slave-lkn4 krb5kdc[2786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513180, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:46:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:46:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:46:20.141896 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43377
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43377
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:46:20.249951 2799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:46:20.250291 2799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:46:20.250331 2799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:46:20.254364 2799 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:46:20.254431 2799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:46:20.254449 2799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:46:20.254464 2799 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:46:20.254478 2799 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:46:20.259770 2799 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43377
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2799
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:46:20.260780 2799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:46:20.261734 2799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 12:46:20.270467 2799 server_base.cc:1034] running on GCE node
W20260927 12:46:20.270491 2807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:20.270527 2805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:20.270731 2804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:20.271307 2799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:46:20.272347 2799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:46:20.273507 2799 hybrid_clock.cc:648] HybridClock initialized: now 1790513180273483 us; error 40 us; skew 500 ppm
Sep 27 12:46:20 dist-test-slave-lkn4 krb5kdc[2786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513180, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:46:20.276144 2799 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:46:20.277031 2799 webserver.cc:460] Webserver started at http://127.0.98.62:37495/ using document root <none> and password file <none>
I20260927 12:46:20.277587 2799 fs_manager.cc:359] Metadata directory not provided
I20260927 12:46:20.277650 2799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:46:20.277823 2799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:46:20.279883 2799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/minicluster-data/master-0/data/instance:
uuid: "1f5f105626d448cba5cce84f4b84966a"
format_stamp: "Formatted at 2026-09-27 12:46:20 on dist-test-slave-lkn4"
server_key: "1cfa9ba75ae0bfdd6a7851ee5acaeea0"
server_key_iv: "8cd08d517b79661115ebfb9b011cf0f7"
server_key_version: "encryptionkey@0"
I20260927 12:46:20.280385 2799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/minicluster-data/master-0/wal/instance:
uuid: "1f5f105626d448cba5cce84f4b84966a"
format_stamp: "Formatted at 2026-09-27 12:46:20 on dist-test-slave-lkn4"
server_key: "1cfa9ba75ae0bfdd6a7851ee5acaeea0"
server_key_iv: "8cd08d517b79661115ebfb9b011cf0f7"
server_key_version: "encryptionkey@0"
I20260927 12:46:20.284176 2799 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 12:46:20.286731 2814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:20.288034 2799 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:46:20.288163 2799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/minicluster-data/master-0/wal
uuid: "1f5f105626d448cba5cce84f4b84966a"
format_stamp: "Formatted at 2026-09-27 12:46:20 on dist-test-slave-lkn4"
server_key: "1cfa9ba75ae0bfdd6a7851ee5acaeea0"
server_key_iv: "8cd08d517b79661115ebfb9b011cf0f7"
server_key_version: "encryptionkey@0"
I20260927 12:46:20.288268 2799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:46:20.302898 2799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:46:20.306218 2799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:46:20.306413 2799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:46:20.313446 2799 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43377
I20260927 12:46:20.313457 2866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43377 every 8 connection(s)
I20260927 12:46:20.314522 2799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:46:20.317516 2867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:46:20.317920 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 2799
I20260927 12:46:20.318018 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/minicluster-data/master-0/wal/instance
I20260927 12:46:20.318285 392 external_mini_cluster.cc:1428] Setting key 36d0b18d70ca95f740527bc470e0c48a
I20260927 12:46:20.324802 2867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f5f105626d448cba5cce84f4b84966a: Bootstrap starting.
Sep 27 12:46:20 dist-test-slave-lkn4 krb5kdc[2786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513180, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:46:20.327724 2867 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f5f105626d448cba5cce84f4b84966a: Neither blocks nor log segments found. Creating new log.
I20260927 12:46:20.328635 2867 log.cc:824] T 00000000000000000000000000000000 P 1f5f105626d448cba5cce84f4b84966a: Log is configured to *not* fsync() on all Append() calls
I20260927 12:46:20.331301 2867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f5f105626d448cba5cce84f4b84966a: No bootstrap required, opened a new log
I20260927 12:46:20.333572 2870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:20.319656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47996 (local address 127.0.98.62:43377)
0927 12:46:20.320060 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:20.320075 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:20.320110 (+ 35us) server_negotiation.cc:407] Connection header received
0927 12:46:20.320791 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:20.320809 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:20.321157 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:20.321451 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:20.322426 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:20.323490 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:20.324798 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:20.325374 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:20.325510 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:20.328435 (+ 2925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:20.328441 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:20.328456 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:20.328496 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:20.330529 (+ 2033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:20.331044 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:20.331051 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:20.331058 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:20.331207 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:20.331497 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:20.331501 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:20.331502 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:20.331966 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:20.332154 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:20.332464 (+ 310us) server_negotiation.cc:299] Negotiation successful
0927 12:46:20.332783 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":122,"threads_started":1}
I20260927 12:46:20.335173 2867 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f5f105626d448cba5cce84f4b84966a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f5f105626d448cba5cce84f4b84966a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43377 } }
I20260927 12:46:20.335419 2867 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f5f105626d448cba5cce84f4b84966a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:46:20.335486 2867 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f5f105626d448cba5cce84f4b84966a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f5f105626d448cba5cce84f4b84966a, State: Initialized, Role: FOLLOWER
I20260927 12:46:20.335953 2867 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f5f105626d448cba5cce84f4b84966a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f5f105626d448cba5cce84f4b84966a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43377 } }
I20260927 12:46:20.336091 2867 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f5f105626d448cba5cce84f4b84966a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:46:20.336154 2867 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f5f105626d448cba5cce84f4b84966a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:46:20.336248 2867 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f5f105626d448cba5cce84f4b84966a [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:46:20.337182 2867 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f5f105626d448cba5cce84f4b84966a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f5f105626d448cba5cce84f4b84966a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43377 } }
I20260927 12:46:20.337520 2867 leader_election.cc:302] T 00000000000000000000000000000000 P 1f5f105626d448cba5cce84f4b84966a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f5f105626d448cba5cce84f4b84966a; no voters:
I20260927 12:46:20.337821 2867 leader_election.cc:288] T 00000000000000000000000000000000 P 1f5f105626d448cba5cce84f4b84966a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:46:20.337934 2872 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f5f105626d448cba5cce84f4b84966a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:46:20.338195 2872 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f5f105626d448cba5cce84f4b84966a [term 1 LEADER]: Becoming Leader. State: Replica: 1f5f105626d448cba5cce84f4b84966a, State: Running, Role: LEADER
I20260927 12:46:20.338557 2872 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f5f105626d448cba5cce84f4b84966a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f5f105626d448cba5cce84f4b84966a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43377 } }
I20260927 12:46:20.339051 2867 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f5f105626d448cba5cce84f4b84966a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:46:20.340093 2874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f5f105626d448cba5cce84f4b84966a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f5f105626d448cba5cce84f4b84966a. Latest consensus state: current_term: 1 leader_uuid: "1f5f105626d448cba5cce84f4b84966a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f5f105626d448cba5cce84f4b84966a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43377 } } }
I20260927 12:46:20.340219 2874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f5f105626d448cba5cce84f4b84966a [sys.catalog]: This master's current role is: LEADER
I20260927 12:46:20.340613 2881 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:46:20.340574 2873 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f5f105626d448cba5cce84f4b84966a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f5f105626d448cba5cce84f4b84966a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f5f105626d448cba5cce84f4b84966a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43377 } } }
I20260927 12:46:20.340766 2873 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f5f105626d448cba5cce84f4b84966a [sys.catalog]: This master's current role is: LEADER
I20260927 12:46:20.343555 2881 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:46:20.348994 2881 catalog_manager.cc:1347] Generated new cluster ID: 0a85fbd01c7d47b09160274f360c7cd6
I20260927 12:46:20.349073 2881 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:46:20.359918 2881 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:46:20.361304 2881 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:46:20.377708 2881 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f5f105626d448cba5cce84f4b84966a: Generated new TSK 0
I20260927 12:46:20.378690 2881 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 12:46:20.418460 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43377
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:46:20.530264 2893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:46:20.530565 2893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:46:20.530625 2893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:46:20.534751 2893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:46:20.534822 2893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:46:20.534904 2893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:46:20.540395 2893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:46:20.541472 2893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:46:20.542527 2893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:46:20.551986 2901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:20.552054 2893 server_base.cc:1034] running on GCE node
W20260927 12:46:20.551981 2898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:20.551976 2899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:20.552606 2893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:46:20.553326 2893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:46:20.554509 2893 hybrid_clock.cc:648] HybridClock initialized: now 1790513180554489 us; error 29 us; skew 500 ppm
Sep 27 12:46:20 dist-test-slave-lkn4 krb5kdc[2786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513180, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:46:20.557331 2893 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 12:46:20.558293 2893 webserver.cc:460] Webserver started at http://127.0.98.1:35531/ using document root <none> and password file <none>
I20260927 12:46:20.558844 2893 fs_manager.cc:359] Metadata directory not provided
I20260927 12:46:20.558904 2893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:46:20.559101 2893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:46:20.561208 2893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-0/data/instance:
uuid: "693decf375384a4ab501f2be3675c0ef"
format_stamp: "Formatted at 2026-09-27 12:46:20 on dist-test-slave-lkn4"
server_key: "984e42936520e94f4ed77666021cfd2e"
server_key_iv: "fe03c8a18d9023639a335e71abef9c14"
server_key_version: "encryptionkey@0"
I20260927 12:46:20.561755 2893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-0/wal/instance:
uuid: "693decf375384a4ab501f2be3675c0ef"
format_stamp: "Formatted at 2026-09-27 12:46:20 on dist-test-slave-lkn4"
server_key: "984e42936520e94f4ed77666021cfd2e"
server_key_iv: "fe03c8a18d9023639a335e71abef9c14"
server_key_version: "encryptionkey@0"
I20260927 12:46:20.565491 2893 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260927 12:46:20.568192 2908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:20.569458 2893 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 12:46:20.569561 2893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-0/wal
uuid: "693decf375384a4ab501f2be3675c0ef"
format_stamp: "Formatted at 2026-09-27 12:46:20 on dist-test-slave-lkn4"
server_key: "984e42936520e94f4ed77666021cfd2e"
server_key_iv: "fe03c8a18d9023639a335e71abef9c14"
server_key_version: "encryptionkey@0"
I20260927 12:46:20.569674 2893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:46:20.592422 2893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:46:20.595670 2893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:46:20.595850 2893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:46:20.596437 2893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:46:20.597276 2893 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:46:20.597322 2893 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:20.597362 2893 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:46:20.597391 2893 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:20.606014 2893 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37619
I20260927 12:46:20.606128 3021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37619 every 8 connection(s)
I20260927 12:46:20.607014 2893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:46:20.615361 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 2893
I20260927 12:46:20.615474 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:46:20.615949 392 external_mini_cluster.cc:1428] Setting key b26468b94f0ac36564fd5c4c2836d704
Sep 27 12:46:20 dist-test-slave-lkn4 krb5kdc[2786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513180, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:46:20.621222 2870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:20.609037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38421 (local address 127.0.98.62:43377)
0927 12:46:20.609230 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:20.609234 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:20.610197 (+ 963us) server_negotiation.cc:407] Connection header received
0927 12:46:20.611630 (+ 1433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:20.611634 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:20.611692 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:20.611776 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:20.613003 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:20.613530 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:20.614752 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:20.614992 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:20.615094 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:20.617776 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:20.617780 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:20.617782 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:20.617806 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:20.619142 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:20.619692 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:20.619696 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:20.619698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:20.619776 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:20.620093 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:20.620096 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:20.620098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:20.620286 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:20.620370 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:20.620890 (+ 520us) server_negotiation.cc:299] Negotiation successful
0927 12:46:20.621022 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 12:46:20.622092 3024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:20.609427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43377 (local address 127.0.98.1:38421)
0927 12:46:20.609990 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:20.610035 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:20.611408 (+ 1373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:20.611965 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:20.611972 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:20.612342 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:20.612852 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:20.612865 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:20.613662 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:20.613665 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:20.614619 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:20.614629 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:20.615147 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:20.615151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:20.615323 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:20.615903 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:20.615920 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:20.617606 (+ 1686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:20.619294 (+ 1688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:20.619301 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:20.619320 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:20.619598 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:20.619858 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:20.619862 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:20.619864 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:20.620009 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:20.620380 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:20.620388 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:20.620650 (+ 262us) client_negotiation.cc:769] Sending connection context
0927 12:46:20.620837 (+ 187us) client_negotiation.cc:240] Negotiation successful
0927 12:46:20.621183 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":141,"threads_started":1}
I20260927 12:46:20.623409 3022 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43377
I20260927 12:46:20.623638 3022 heartbeater.cc:464] Registering TS with master...
I20260927 12:46:20.624162 3022 heartbeater.cc:511] Master 127.0.98.62:43377 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 12:46:20.625636 2831 ts_manager.cc:194] Registered new tserver with Master: 693decf375384a4ab501f2be3675c0ef (127.0.98.1:37619)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 12:46:20.626952 2831 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38421
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 12:46:20.642817 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43377
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:46:20.757522 3027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:46:20.757827 3027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:46:20.757871 3027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:46:20.762117 3027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:46:20.762193 3027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:46:20.762259 3027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:46:20.767704 3027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:46:20.768818 3027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:46:20.769830 3027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:46:20.780010 3035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:20.780062 3032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:20.780113 3033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:20.780260 3027 server_base.cc:1034] running on GCE node
I20260927 12:46:20.780690 3027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:46:20.781388 3027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:46:20.782557 3027 hybrid_clock.cc:648] HybridClock initialized: now 1790513180782520 us; error 49 us; skew 500 ppm
Sep 27 12:46:20 dist-test-slave-lkn4 krb5kdc[2786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513180, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:46:20.785485 3027 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 12:46:20.786473 3027 webserver.cc:460] Webserver started at http://127.0.98.2:40523/ using document root <none> and password file <none>
I20260927 12:46:20.787017 3027 fs_manager.cc:359] Metadata directory not provided
I20260927 12:46:20.787074 3027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:46:20.787258 3027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:46:20.789402 3027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-1/data/instance:
uuid: "98c8d7bf85d44cb1b700f3a29334b83e"
format_stamp: "Formatted at 2026-09-27 12:46:20 on dist-test-slave-lkn4"
server_key: "2878911addd246114f380da2bbf8295b"
server_key_iv: "65a062582e4718c8fa60c646e17f3253"
server_key_version: "encryptionkey@0"
I20260927 12:46:20.789937 3027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-1/wal/instance:
uuid: "98c8d7bf85d44cb1b700f3a29334b83e"
format_stamp: "Formatted at 2026-09-27 12:46:20 on dist-test-slave-lkn4"
server_key: "2878911addd246114f380da2bbf8295b"
server_key_iv: "65a062582e4718c8fa60c646e17f3253"
server_key_version: "encryptionkey@0"
I20260927 12:46:20.794009 3027 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260927 12:46:20.796765 3042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:20.798234 3027 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 12:46:20.798362 3027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-1/wal
uuid: "98c8d7bf85d44cb1b700f3a29334b83e"
format_stamp: "Formatted at 2026-09-27 12:46:20 on dist-test-slave-lkn4"
server_key: "2878911addd246114f380da2bbf8295b"
server_key_iv: "65a062582e4718c8fa60c646e17f3253"
server_key_version: "encryptionkey@0"
I20260927 12:46:20.798476 3027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:46:20.821225 3027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:46:20.824779 3027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:46:20.824966 3027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:46:20.825594 3027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:46:20.826534 3027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:46:20.826583 3027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:20.826637 3027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:46:20.826665 3027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:20.835632 3027 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39249
I20260927 12:46:20.835657 3155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39249 every 8 connection(s)
I20260927 12:46:20.836623 3027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:46:20.839708 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 3027
I20260927 12:46:20.839834 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:46:20.840169 392 external_mini_cluster.cc:1428] Setting key 0252bb30f7f86c3b6512278891d20371
Sep 27 12:46:20 dist-test-slave-lkn4 krb5kdc[2786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513180, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 12:46:20.852017 2870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:20.838503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34015 (local address 127.0.98.62:43377)
0927 12:46:20.838666 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:20.838671 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:20.839649 (+ 978us) server_negotiation.cc:407] Connection header received
0927 12:46:20.841462 (+ 1813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:20.841466 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:20.841510 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:20.841581 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:20.842870 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:20.843487 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:20.844930 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:20.845273 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:20.845385 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:20.847912 (+ 2527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:20.847917 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:20.847920 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:20.847952 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:20.849555 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:20.850134 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:20.850139 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:20.850140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:20.850250 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:20.850596 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:20.850599 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:20.850671 (+ 72us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:20.850886 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:20.850979 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:20.851636 (+ 657us) server_negotiation.cc:299] Negotiation successful
0927 12:46:20.851849 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 12:46:20.853027 3158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:20.838907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43377 (local address 127.0.98.2:34015)
0927 12:46:20.839460 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:20.839524 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:20.841276 (+ 1752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:20.841771 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:20.841779 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:20.842210 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:20.842721 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:20.842737 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:20.843630 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:20.843634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:20.844797 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:20.844807 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:20.845458 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:20.845463 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:20.845663 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:20.846276 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:20.846294 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:20.847721 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:20.849683 (+ 1962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:20.849690 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:20.849708 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:20.849979 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:20.850342 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:20.850345 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:20.850346 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:20.850504 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:20.851053 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:20.851061 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:20.851374 (+ 313us) client_negotiation.cc:769] Sending connection context
0927 12:46:20.851587 (+ 213us) client_negotiation.cc:240] Negotiation successful
0927 12:46:20.851943 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":117,"threads_started":1}
I20260927 12:46:20.854293 3156 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43377
I20260927 12:46:20.854535 3156 heartbeater.cc:464] Registering TS with master...
I20260927 12:46:20.855020 3156 heartbeater.cc:511] Master 127.0.98.62:43377 requested a full tablet report, sending...
I20260927 12:46:20.856108 2831 ts_manager.cc:194] Registered new tserver with Master: 98c8d7bf85d44cb1b700f3a29334b83e (127.0.98.2:39249)
I20260927 12:46:20.856848 2831 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34015
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 12:46:20.869988 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43377
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:46:20.985018 3161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:46:20.985298 3161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:46:20.985342 3161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:46:20.989383 3161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:46:20.989445 3161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:46:20.989511 3161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:46:20.994962 3161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43377
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:46:20.996055 3161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:46:20.997033 3161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:46:21.006515 3167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:21.006564 3166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:21.006529 3169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:21.006830 3161 server_base.cc:1034] running on GCE node
I20260927 12:46:21.007308 3161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:46:21.007951 3161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:46:21.009137 3161 hybrid_clock.cc:648] HybridClock initialized: now 1790513181009119 us; error 39 us; skew 500 ppm
Sep 27 12:46:21 dist-test-slave-lkn4 krb5kdc[2786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513181, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:46:21.011950 3161 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 12:46:21.012861 3161 webserver.cc:460] Webserver started at http://127.0.98.3:40509/ using document root <none> and password file <none>
I20260927 12:46:21.013398 3161 fs_manager.cc:359] Metadata directory not provided
I20260927 12:46:21.013455 3161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:46:21.013635 3161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:46:21.015766 3161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-2/data/instance:
uuid: "362ac9d7f78c40eb850a7fc98d165f3c"
format_stamp: "Formatted at 2026-09-27 12:46:21 on dist-test-slave-lkn4"
server_key: "2ee077243a26b7084de821f89b2f0217"
server_key_iv: "1d630978bd9244c58fc935df91e6e04b"
server_key_version: "encryptionkey@0"
I20260927 12:46:21.016283 3161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-2/wal/instance:
uuid: "362ac9d7f78c40eb850a7fc98d165f3c"
format_stamp: "Formatted at 2026-09-27 12:46:21 on dist-test-slave-lkn4"
server_key: "2ee077243a26b7084de821f89b2f0217"
server_key_iv: "1d630978bd9244c58fc935df91e6e04b"
server_key_version: "encryptionkey@0"
I20260927 12:46:21.020174 3161 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 12:46:21.022677 3176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:21.024099 3161 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:46:21.024197 3161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-2/wal
uuid: "362ac9d7f78c40eb850a7fc98d165f3c"
format_stamp: "Formatted at 2026-09-27 12:46:21 on dist-test-slave-lkn4"
server_key: "2ee077243a26b7084de821f89b2f0217"
server_key_iv: "1d630978bd9244c58fc935df91e6e04b"
server_key_version: "encryptionkey@0"
I20260927 12:46:21.024314 3161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:46:21.039382 3161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:46:21.042734 3161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:46:21.042896 3161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:46:21.043462 3161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:46:21.044268 3161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:46:21.044322 3161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:21.044359 3161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:46:21.044384 3161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:21.052310 3161 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42263
I20260927 12:46:21.052374 3289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42263 every 8 connection(s)
I20260927 12:46:21.053243 3161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:46:21.056573 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 3161
I20260927 12:46:21.056672 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:46:21.056928 392 external_mini_cluster.cc:1428] Setting key 04ca5d0e100c9d2267c20bd2b105283d
Sep 27 12:46:21 dist-test-slave-lkn4 krb5kdc[2786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513181, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:46:21.067173 2870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:21.055217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60605 (local address 127.0.98.62:43377)
0927 12:46:21.055388 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:21.055392 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:21.056273 (+ 881us) server_negotiation.cc:407] Connection header received
0927 12:46:21.057847 (+ 1574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:21.057850 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:21.057916 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:21.057987 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:21.059329 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:21.059853 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:21.060997 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:21.061210 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:21.061296 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:21.063722 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:21.063725 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:21.063727 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:21.063753 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:21.065077 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:21.065580 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:21.065583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:21.065584 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:21.065660 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:21.065980 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:21.065984 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:21.065986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:21.066225 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:21.066323 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:21.066940 (+ 617us) server_negotiation.cc:299] Negotiation successful
0927 12:46:21.067047 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 12:46:21.068130 3292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:21.055612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43377 (local address 127.0.98.3:60605)
0927 12:46:21.056098 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:21.056142 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:21.057620 (+ 1478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:21.058219 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:21.058228 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:21.058601 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:21.059182 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:21.059200 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:21.059967 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:21.059971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:21.060877 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:21.060886 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:21.061347 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:21.061351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:21.061517 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:21.062114 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:21.062131 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:21.063549 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:21.065189 (+ 1640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:21.065196 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:21.065222 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:21.065483 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:21.065745 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:21.065748 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:21.065750 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:21.065895 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:21.066405 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:21.066411 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:21.066674 (+ 263us) client_negotiation.cc:769] Sending connection context
0927 12:46:21.066901 (+ 227us) client_negotiation.cc:240] Negotiation successful
0927 12:46:21.067189 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":137,"threads_started":1}
I20260927 12:46:21.069386 3290 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43377
I20260927 12:46:21.069734 3290 heartbeater.cc:464] Registering TS with master...
I20260927 12:46:21.070304 3290 heartbeater.cc:511] Master 127.0.98.62:43377 requested a full tablet report, sending...
I20260927 12:46:21.071290 2831 ts_manager.cc:194] Registered new tserver with Master: 362ac9d7f78c40eb850a7fc98d165f3c (127.0.98.3:42263)
I20260927 12:46:21.071853 2831 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60605
I20260927 12:46:21.083526 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 12:46:21.096098 2870 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:46:21.094662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48010 (local address 127.0.98.62:43377)
0927 12:46:21.094831 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:21.094836 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:21.094963 (+ 127us) server_negotiation.cc:407] Connection header received
0927 12:46:21.095076 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:21.095078 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:21.095123 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:21.095192 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:21.095962 (+ 770us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48010: BlockingRecv error: recv got EOF from 127.0.0.1:48010 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 12:46:21.097853 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 2893
I20260927 12:46:21.103003 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 3027
I20260927 12:46:21.108069 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 3161
I20260927 12:46:21.112638 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 2799
2026-09-27T12:46:21Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2060 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790513161160093-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:46:21 dist-test-slave-lkn4 krb5kdc[3302](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:46:21 dist-test-slave-lkn4 krb5kdc[3302](info): set up 2 sockets
Sep 27 12:46:21 dist-test-slave-lkn4 krb5kdc[3302](info): commencing operation
krb5kdc: starting...
W20260927 12:46:22.073866 392 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790513161160093-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790513161160093-392-0/krb5kdc/test-user.keytab.
Sep 27 12:46:22 dist-test-slave-lkn4 krb5kdc[3302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:46:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:46:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:46:22.194192 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790513161160093-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790513161160093-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790513161160093-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38867
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38867
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:46:22.303421 3315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:46:22.303726 3315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:46:22.303767 3315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:46:22.307847 3315 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:46:22.307924 3315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:46:22.307943 3315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:46:22.307958 3315 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:46:22.307972 3315 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:46:22.313361 3315 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790513161160093-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790513161160093-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790513161160093-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:46:22.314568 3315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:46:22.315573 3315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:46:22.324004 3321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:22.324299 3315 server_base.cc:1034] running on GCE node
W20260927 12:46:22.325273 3320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:22.325910 3323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:22.326536 3315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:46:22.327714 3315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:46:22.328879 3315 hybrid_clock.cc:648] HybridClock initialized: now 1790513182328857 us; error 37 us; skew 500 ppm
Sep 27 12:46:22 dist-test-slave-lkn4 krb5kdc[3302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513182, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:46:22.331692 3315 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:46:22.332654 3315 webserver.cc:460] Webserver started at http://127.0.98.62:35007/ using document root <none> and password file <none>
I20260927 12:46:22.333217 3315 fs_manager.cc:359] Metadata directory not provided
I20260927 12:46:22.333281 3315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:46:22.333477 3315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:46:22.335626 3315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790513161160093-392-0/minicluster-data/master-0/data/instance:
uuid: "b8c71bb4894e4240867b0eddb20ecf56"
format_stamp: "Formatted at 2026-09-27 12:46:22 on dist-test-slave-lkn4"
server_key: "3d5c6accd9e53f31e81410dafbe96867"
server_key_iv: "e868b87f8064342d8d09efea3c9ccf8c"
server_key_version: "encryptionkey@0"
I20260927 12:46:22.336176 3315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790513161160093-392-0/minicluster-data/master-0/wal/instance:
uuid: "b8c71bb4894e4240867b0eddb20ecf56"
format_stamp: "Formatted at 2026-09-27 12:46:22 on dist-test-slave-lkn4"
server_key: "3d5c6accd9e53f31e81410dafbe96867"
server_key_iv: "e868b87f8064342d8d09efea3c9ccf8c"
server_key_version: "encryptionkey@0"
I20260927 12:46:22.339984 3315 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 12:46:22.342525 3330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:22.343900 3315 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:46:22.344022 3315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790513161160093-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790513161160093-392-0/minicluster-data/master-0/wal
uuid: "b8c71bb4894e4240867b0eddb20ecf56"
format_stamp: "Formatted at 2026-09-27 12:46:22 on dist-test-slave-lkn4"
server_key: "3d5c6accd9e53f31e81410dafbe96867"
server_key_iv: "e868b87f8064342d8d09efea3c9ccf8c"
server_key_version: "encryptionkey@0"
I20260927 12:46:22.344121 3315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790513161160093-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790513161160093-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790513161160093-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:46:22.364586 3315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:46:22.367755 3315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:46:22.367939 3315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:46:22.374758 3315 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38867
I20260927 12:46:22.375023 3382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38867 every 8 connection(s)
I20260927 12:46:22.375722 3315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:46:22.378568 3383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:46:22.380651 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 3315
I20260927 12:46:22.380767 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790513161160093-392-0/minicluster-data/master-0/wal/instance
I20260927 12:46:22.381048 392 external_mini_cluster.cc:1428] Setting key 177640e6f3cf151bc23e3af0d1c3424d
I20260927 12:46:22.385429 3383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8c71bb4894e4240867b0eddb20ecf56: Bootstrap starting.
I20260927 12:46:22.388017 3383 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8c71bb4894e4240867b0eddb20ecf56: Neither blocks nor log segments found. Creating new log.
I20260927 12:46:22.388876 3383 log.cc:824] T 00000000000000000000000000000000 P b8c71bb4894e4240867b0eddb20ecf56: Log is configured to *not* fsync() on all Append() calls
Sep 27 12:46:22 dist-test-slave-lkn4 krb5kdc[3302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:46:22.390894 3383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8c71bb4894e4240867b0eddb20ecf56: No bootstrap required, opened a new log
I20260927 12:46:22.395844 3386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:22.382402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41894 (local address 127.0.98.62:38867)
0927 12:46:22.382797 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:22.382817 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:22.382858 (+ 41us) server_negotiation.cc:407] Connection header received
0927 12:46:22.383596 (+ 738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:22.383620 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:22.384042 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:22.384413 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:22.385172 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:22.386571 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:22.387446 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:22.388097 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:22.388264 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:22.390614 (+ 2350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:22.390620 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:22.390637 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:22.390676 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:22.392588 (+ 1912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:22.393043 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:22.393049 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:22.393056 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:22.393219 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:22.393463 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:22.393466 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:22.393468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:22.393964 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:22.394319 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:22.394481 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 12:46:22.394816 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":95,"threads_started":1}
I20260927 12:46:22.395682 3383 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8c71bb4894e4240867b0eddb20ecf56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8c71bb4894e4240867b0eddb20ecf56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38867 } }
I20260927 12:46:22.395947 3383 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8c71bb4894e4240867b0eddb20ecf56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:46:22.395989 3383 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8c71bb4894e4240867b0eddb20ecf56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8c71bb4894e4240867b0eddb20ecf56, State: Initialized, Role: FOLLOWER
I20260927 12:46:22.396477 3383 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8c71bb4894e4240867b0eddb20ecf56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8c71bb4894e4240867b0eddb20ecf56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38867 } }
I20260927 12:46:22.396637 3383 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8c71bb4894e4240867b0eddb20ecf56 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:46:22.396690 3383 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8c71bb4894e4240867b0eddb20ecf56 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:46:22.396770 3383 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8c71bb4894e4240867b0eddb20ecf56 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:46:22.398002 3383 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8c71bb4894e4240867b0eddb20ecf56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8c71bb4894e4240867b0eddb20ecf56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38867 } }
I20260927 12:46:22.398373 3383 leader_election.cc:302] T 00000000000000000000000000000000 P b8c71bb4894e4240867b0eddb20ecf56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8c71bb4894e4240867b0eddb20ecf56; no voters:
I20260927 12:46:22.398694 3383 leader_election.cc:288] T 00000000000000000000000000000000 P b8c71bb4894e4240867b0eddb20ecf56 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:46:22.398794 3388 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8c71bb4894e4240867b0eddb20ecf56 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:46:22.398988 3388 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8c71bb4894e4240867b0eddb20ecf56 [term 1 LEADER]: Becoming Leader. State: Replica: b8c71bb4894e4240867b0eddb20ecf56, State: Running, Role: LEADER
I20260927 12:46:22.399011 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:46:22.399353 3388 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8c71bb4894e4240867b0eddb20ecf56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8c71bb4894e4240867b0eddb20ecf56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38867 } }
I20260927 12:46:22.400054 3383 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8c71bb4894e4240867b0eddb20ecf56 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:46:22.400974 3390 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8c71bb4894e4240867b0eddb20ecf56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8c71bb4894e4240867b0eddb20ecf56. Latest consensus state: current_term: 1 leader_uuid: "b8c71bb4894e4240867b0eddb20ecf56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8c71bb4894e4240867b0eddb20ecf56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38867 } } }
I20260927 12:46:22.400981 3389 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8c71bb4894e4240867b0eddb20ecf56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b8c71bb4894e4240867b0eddb20ecf56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8c71bb4894e4240867b0eddb20ecf56" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38867 } } }
I20260927 12:46:22.401113 3390 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8c71bb4894e4240867b0eddb20ecf56 [sys.catalog]: This master's current role is: LEADER
I20260927 12:46:22.401113 3389 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8c71bb4894e4240867b0eddb20ecf56 [sys.catalog]: This master's current role is: LEADER
I20260927 12:46:22.401516 3398 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:46:22.404778 3398 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:46:22.410879 3398 catalog_manager.cc:1347] Generated new cluster ID: 1362cb046883468c9414fdbbba337b98
I20260927 12:46:22.410952 3398 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:46:22.428087 3386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:22.422376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41900 (local address 127.0.98.62:38867)
0927 12:46:22.422498 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:22.422503 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:22.422740 (+ 237us) server_negotiation.cc:407] Connection header received
0927 12:46:22.422876 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:22.422879 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:22.422927 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:22.423020 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:22.423612 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:22.424390 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:22.425576 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:22.425915 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:22.426004 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:22.426955 (+ 951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:22.426959 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:22.426962 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:46:22.427617 (+ 655us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:22.427653 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:46:22.427672 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:22.427763 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:22.427868 (+ 105us) server_negotiation.cc:299] Negotiation successful
0927 12:46:22.427934 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 12:46:22.429919 3398 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:46:22.430872 3398 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:46:22.437889 3398 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8c71bb4894e4240867b0eddb20ecf56: Generated new TSK 0
I20260927 12:46:22.438594 3398 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:46:22.444428 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 3315
2026-09-27T12:46:22Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1329 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:46:22 dist-test-slave-lkn4 krb5kdc[3416](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:46:22 dist-test-slave-lkn4 krb5kdc[3416](info): set up 2 sockets
Sep 27 12:46:22 dist-test-slave-lkn4 krb5kdc[3416](info): commencing operation
krb5kdc: starting...
W20260927 12:46:23.401882 392 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/krb5kdc/test-user.keytab.
Sep 27 12:46:23 dist-test-slave-lkn4 krb5kdc[3416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513183, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:46:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:46:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:46:23.522905 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40307
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40307
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:46:23.632747 3429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:46:23.633054 3429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:46:23.633112 3429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:46:23.637202 3429 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:46:23.637293 3429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:46:23.637316 3429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:46:23.637344 3429 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:46:23.637375 3429 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:46:23.642879 3429 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40307
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:46:23.643885 3429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:46:23.644851 3429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:46:23.653163 3437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:23.653208 3429 server_base.cc:1034] running on GCE node
W20260927 12:46:23.653188 3434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:23.653163 3435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:23.653903 3429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:46:23.655097 3429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:46:23.656294 3429 hybrid_clock.cc:648] HybridClock initialized: now 1790513183656277 us; error 33 us; skew 500 ppm
Sep 27 12:46:23 dist-test-slave-lkn4 krb5kdc[3416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513183, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:46:23.659154 3429 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:46:23.660106 3429 webserver.cc:460] Webserver started at http://127.0.98.62:44925/ using document root <none> and password file <none>
I20260927 12:46:23.660627 3429 fs_manager.cc:359] Metadata directory not provided
I20260927 12:46:23.660696 3429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:46:23.660856 3429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:46:23.662941 3429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/minicluster-data/master-0/data/instance:
uuid: "56207af12d1c491190600f88d0cc782e"
format_stamp: "Formatted at 2026-09-27 12:46:23 on dist-test-slave-lkn4"
server_key: "bac709a5ed2656e28a209e1d48026068"
server_key_iv: "4ffbb13b18c86a589534673aecbaddcd"
server_key_version: "encryptionkey@0"
I20260927 12:46:23.663468 3429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/minicluster-data/master-0/wal/instance:
uuid: "56207af12d1c491190600f88d0cc782e"
format_stamp: "Formatted at 2026-09-27 12:46:23 on dist-test-slave-lkn4"
server_key: "bac709a5ed2656e28a209e1d48026068"
server_key_iv: "4ffbb13b18c86a589534673aecbaddcd"
server_key_version: "encryptionkey@0"
I20260927 12:46:23.667158 3429 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 12:46:23.669445 3444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:23.670822 3429 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:46:23.670951 3429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/minicluster-data/master-0/wal
uuid: "56207af12d1c491190600f88d0cc782e"
format_stamp: "Formatted at 2026-09-27 12:46:23 on dist-test-slave-lkn4"
server_key: "bac709a5ed2656e28a209e1d48026068"
server_key_iv: "4ffbb13b18c86a589534673aecbaddcd"
server_key_version: "encryptionkey@0"
I20260927 12:46:23.671056 3429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:46:23.686280 3429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:46:23.689424 3429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:46:23.689603 3429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:46:23.696559 3429 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40307
I20260927 12:46:23.696822 3496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40307 every 8 connection(s)
I20260927 12:46:23.697604 3429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:46:23.699081 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 3429
I20260927 12:46:23.699198 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/minicluster-data/master-0/wal/instance
I20260927 12:46:23.699502 392 external_mini_cluster.cc:1428] Setting key 90ed238fc70c7cc8a00ab43762284a42
I20260927 12:46:23.700912 3497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:46:23.707343 3497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56207af12d1c491190600f88d0cc782e: Bootstrap starting.
Sep 27 12:46:23 dist-test-slave-lkn4 krb5kdc[3416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513183, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:46:23.709786 3497 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56207af12d1c491190600f88d0cc782e: Neither blocks nor log segments found. Creating new log.
I20260927 12:46:23.710541 3497 log.cc:824] T 00000000000000000000000000000000 P 56207af12d1c491190600f88d0cc782e: Log is configured to *not* fsync() on all Append() calls
I20260927 12:46:23.712438 3497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56207af12d1c491190600f88d0cc782e: No bootstrap required, opened a new log
I20260927 12:46:23.713677 3500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:23.700956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44344 (local address 127.0.98.62:40307)
0927 12:46:23.701429 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:23.701445 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:23.701482 (+ 37us) server_negotiation.cc:407] Connection header received
0927 12:46:23.702190 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:23.702209 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:23.702554 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:23.702852 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:23.703716 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:23.704775 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:23.705926 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:23.706439 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:23.706566 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:23.709206 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:23.709211 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:23.709248 (+ 37us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:23.709279 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:23.710977 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:23.711396 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:23.711401 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:23.711407 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:23.711516 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:23.711781 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:23.711784 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:23.711786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:23.712177 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:23.712363 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:23.712671 (+ 308us) server_negotiation.cc:299] Negotiation successful
0927 12:46:23.712952 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":167,"threads_started":1}
I20260927 12:46:23.716413 3497 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56207af12d1c491190600f88d0cc782e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56207af12d1c491190600f88d0cc782e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40307 } }
I20260927 12:46:23.716640 3497 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56207af12d1c491190600f88d0cc782e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:46:23.716704 3497 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56207af12d1c491190600f88d0cc782e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56207af12d1c491190600f88d0cc782e, State: Initialized, Role: FOLLOWER
I20260927 12:46:23.717090 3497 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56207af12d1c491190600f88d0cc782e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56207af12d1c491190600f88d0cc782e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40307 } }
I20260927 12:46:23.717224 3497 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56207af12d1c491190600f88d0cc782e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:46:23.717285 3497 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56207af12d1c491190600f88d0cc782e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:46:23.717399 3497 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56207af12d1c491190600f88d0cc782e [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:46:23.718361 3497 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56207af12d1c491190600f88d0cc782e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56207af12d1c491190600f88d0cc782e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40307 } }
I20260927 12:46:23.718677 3497 leader_election.cc:302] T 00000000000000000000000000000000 P 56207af12d1c491190600f88d0cc782e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56207af12d1c491190600f88d0cc782e; no voters:
I20260927 12:46:23.718959 3497 leader_election.cc:288] T 00000000000000000000000000000000 P 56207af12d1c491190600f88d0cc782e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:46:23.719084 3502 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56207af12d1c491190600f88d0cc782e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:46:23.719298 3502 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56207af12d1c491190600f88d0cc782e [term 1 LEADER]: Becoming Leader. State: Replica: 56207af12d1c491190600f88d0cc782e, State: Running, Role: LEADER
I20260927 12:46:23.719633 3502 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56207af12d1c491190600f88d0cc782e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56207af12d1c491190600f88d0cc782e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40307 } }
I20260927 12:46:23.720088 3497 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56207af12d1c491190600f88d0cc782e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:46:23.721146 3504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56207af12d1c491190600f88d0cc782e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56207af12d1c491190600f88d0cc782e. Latest consensus state: current_term: 1 leader_uuid: "56207af12d1c491190600f88d0cc782e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56207af12d1c491190600f88d0cc782e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40307 } } }
I20260927 12:46:23.721271 3504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56207af12d1c491190600f88d0cc782e [sys.catalog]: This master's current role is: LEADER
I20260927 12:46:23.721490 3503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56207af12d1c491190600f88d0cc782e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56207af12d1c491190600f88d0cc782e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56207af12d1c491190600f88d0cc782e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40307 } } }
I20260927 12:46:23.721571 3503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56207af12d1c491190600f88d0cc782e [sys.catalog]: This master's current role is: LEADER
I20260927 12:46:23.721640 3511 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:46:23.724762 3511 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:46:23.730356 3511 catalog_manager.cc:1347] Generated new cluster ID: e84bf68258834d81a6e238c603d471e1
I20260927 12:46:23.730427 3511 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:46:23.751430 3511 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:46:23.752390 3511 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:46:23.759975 3511 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 56207af12d1c491190600f88d0cc782e: Generated new TSK 0
I20260927 12:46:23.760787 3511 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 12:46:23.801908 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40307
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:46:23.913712 3523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:46:23.914007 3523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:46:23.914090 3523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:46:23.918206 3523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:46:23.918279 3523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:46:23.918360 3523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:46:23.923868 3523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:46:23.924998 3523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:46:23.926070 3523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:46:23.935635 3529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:23.935653 3528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:23.935725 3523 server_base.cc:1034] running on GCE node
W20260927 12:46:23.935635 3531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:23.936272 3523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:46:23.936982 3523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:46:23.938138 3523 hybrid_clock.cc:648] HybridClock initialized: now 1790513183938116 us; error 33 us; skew 500 ppm
Sep 27 12:46:23 dist-test-slave-lkn4 krb5kdc[3416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513183, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:46:23.940872 3523 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 12:46:23.941769 3523 webserver.cc:460] Webserver started at http://127.0.98.1:44797/ using document root <none> and password file <none>
I20260927 12:46:23.942333 3523 fs_manager.cc:359] Metadata directory not provided
I20260927 12:46:23.942394 3523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:46:23.942560 3523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:46:23.944593 3523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-0/data/instance:
uuid: "8574f02ae8044d1ea39fb599d6358144"
format_stamp: "Formatted at 2026-09-27 12:46:23 on dist-test-slave-lkn4"
server_key: "3b8a367c749288328a859b7d765290a6"
server_key_iv: "337c8fc4221d86afbfb01e863b389ee4"
server_key_version: "encryptionkey@0"
I20260927 12:46:23.945134 3523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8574f02ae8044d1ea39fb599d6358144"
format_stamp: "Formatted at 2026-09-27 12:46:23 on dist-test-slave-lkn4"
server_key: "3b8a367c749288328a859b7d765290a6"
server_key_iv: "337c8fc4221d86afbfb01e863b389ee4"
server_key_version: "encryptionkey@0"
I20260927 12:46:23.949492 3523 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 12:46:23.951956 3538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:23.953265 3523 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:46:23.953379 3523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-0/wal
uuid: "8574f02ae8044d1ea39fb599d6358144"
format_stamp: "Formatted at 2026-09-27 12:46:23 on dist-test-slave-lkn4"
server_key: "3b8a367c749288328a859b7d765290a6"
server_key_iv: "337c8fc4221d86afbfb01e863b389ee4"
server_key_version: "encryptionkey@0"
I20260927 12:46:23.953483 3523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:46:23.975049 3523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:46:23.979290 3523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:46:23.979527 3523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:46:23.980299 3523 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:46:23.981431 3523 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:46:23.981503 3523 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:23.981571 3523 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:46:23.981606 3523 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:23.991282 3523 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33839
I20260927 12:46:23.991346 3651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33839 every 8 connection(s)
I20260927 12:46:23.992272 3523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:46:23.998461 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 3523
I20260927 12:46:23.998572 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:46:23.998835 392 external_mini_cluster.cc:1428] Setting key 11a01c565eb8a218a0afb1575c78ba8c
Sep 27 12:46:24 dist-test-slave-lkn4 krb5kdc[3416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513183, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 12:46:24.009109 3500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:23.994445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33471 (local address 127.0.98.62:40307)
0927 12:46:23.994577 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:23.994581 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:23.995499 (+ 918us) server_negotiation.cc:407] Connection header received
0927 12:46:23.996934 (+ 1435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:23.996937 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:23.996984 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:23.997059 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:23.998474 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:23.999189 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:24.000488 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:24.000935 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:24.001066 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:24.005359 (+ 4293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:24.005364 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:24.005366 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:24.005393 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:24.006881 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:24.007531 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:24.007533 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:24.007535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:24.007615 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:24.007996 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:24.007999 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:24.008000 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:24.008216 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:24.008300 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:24.008863 (+ 563us) server_negotiation.cc:299] Negotiation successful
0927 12:46:24.008979 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 12:46:24.010033 3654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:23.994875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40307 (local address 127.0.98.1:33471)
0927 12:46:23.995338 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:23.995384 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:23.996702 (+ 1318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:23.997221 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:23.997229 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:23.997619 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:23.998283 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:23.998306 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:23.999331 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:23.999334 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:24.000351 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:24.000363 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:24.001142 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:24.001147 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:24.001370 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:24.003150 (+ 1780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:24.003173 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:24.005113 (+ 1940us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:24.007016 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:24.007024 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:24.007044 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:24.007423 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:24.007703 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:24.007706 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:24.007708 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:24.007912 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:24.008315 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:24.008323 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:24.008633 (+ 310us) client_negotiation.cc:769] Sending connection context
0927 12:46:24.008816 (+ 183us) client_negotiation.cc:240] Negotiation successful
0927 12:46:24.009099 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":109,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 12:46:24.011354 3652 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40307
I20260927 12:46:24.011615 3652 heartbeater.cc:464] Registering TS with master...
I20260927 12:46:24.012200 3652 heartbeater.cc:511] Master 127.0.98.62:40307 requested a full tablet report, sending...
I20260927 12:46:24.013854 3461 ts_manager.cc:194] Registered new tserver with Master: 8574f02ae8044d1ea39fb599d6358144 (127.0.98.1:33839)
I20260927 12:46:24.015398 3461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33471
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 12:46:24.029412 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40307
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:46:24.139597 3657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:46:24.139878 3657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:46:24.139919 3657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:46:24.143962 3657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:46:24.144027 3657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:46:24.144084 3657 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:46:24.149400 3657 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:46:24.150444 3657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:46:24.151412 3657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:46:24.161170 3662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:24.161193 3665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:24.161182 3663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:24.161406 3657 server_base.cc:1034] running on GCE node
I20260927 12:46:24.161849 3657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:46:24.162509 3657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:46:24.163672 3657 hybrid_clock.cc:648] HybridClock initialized: now 1790513184163651 us; error 33 us; skew 500 ppm
Sep 27 12:46:24 dist-test-slave-lkn4 krb5kdc[3416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513184, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:46:24.171103 3657 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 12:46:24.172057 3657 webserver.cc:460] Webserver started at http://127.0.98.2:36439/ using document root <none> and password file <none>
I20260927 12:46:24.172623 3657 fs_manager.cc:359] Metadata directory not provided
I20260927 12:46:24.172690 3657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:46:24.172943 3657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:46:24.175154 3657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-1/data/instance:
uuid: "698359fb7f9c403cbc6b5cd8bc362562"
format_stamp: "Formatted at 2026-09-27 12:46:24 on dist-test-slave-lkn4"
server_key: "7a14adf60136549a5fd13f3431efbbf9"
server_key_iv: "7a8a38a88ee6820d172628584ded46a0"
server_key_version: "encryptionkey@0"
I20260927 12:46:24.175698 3657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-1/wal/instance:
uuid: "698359fb7f9c403cbc6b5cd8bc362562"
format_stamp: "Formatted at 2026-09-27 12:46:24 on dist-test-slave-lkn4"
server_key: "7a14adf60136549a5fd13f3431efbbf9"
server_key_iv: "7a8a38a88ee6820d172628584ded46a0"
server_key_version: "encryptionkey@0"
I20260927 12:46:24.179404 3657 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 12:46:24.181752 3672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:24.182965 3657 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 12:46:24.183079 3657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-1/wal
uuid: "698359fb7f9c403cbc6b5cd8bc362562"
format_stamp: "Formatted at 2026-09-27 12:46:24 on dist-test-slave-lkn4"
server_key: "7a14adf60136549a5fd13f3431efbbf9"
server_key_iv: "7a8a38a88ee6820d172628584ded46a0"
server_key_version: "encryptionkey@0"
I20260927 12:46:24.183174 3657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:46:24.205602 3657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:46:24.208858 3657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:46:24.209044 3657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:46:24.209656 3657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:46:24.210527 3657 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:46:24.210575 3657 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:24.210628 3657 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:46:24.210654 3657 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:24.219441 3657 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41845
I20260927 12:46:24.219599 3785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41845 every 8 connection(s)
I20260927 12:46:24.220427 3657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:46:24.225837 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 3657
I20260927 12:46:24.225960 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:46:24.226284 392 external_mini_cluster.cc:1428] Setting key 503e87dc2b1c7eb075fb151e1bc591d3
Sep 27 12:46:24 dist-test-slave-lkn4 krb5kdc[3416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513184, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 12:46:24.234817 3500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:24.222435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43397 (local address 127.0.98.62:40307)
0927 12:46:24.222585 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:24.222590 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:24.223527 (+ 937us) server_negotiation.cc:407] Connection header received
0927 12:46:24.224926 (+ 1399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:24.224929 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:24.224980 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:24.225056 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:24.226550 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:24.227156 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:24.228370 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:24.228586 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:24.228673 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:24.231258 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:24.231262 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:24.231264 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:24.231289 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:24.232604 (+ 1315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:24.233127 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:24.233130 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:24.233132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:24.233211 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:24.233530 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:24.233533 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:24.233535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:24.233734 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:24.233822 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:24.234441 (+ 619us) server_negotiation.cc:299] Negotiation successful
0927 12:46:24.234636 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 12:46:24.235677 3788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:24.222891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40307 (local address 127.0.98.2:43397)
0927 12:46:24.223363 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:24.223415 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:24.224699 (+ 1284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:24.225207 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:24.225214 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:24.225613 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:24.226372 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:24.226402 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:24.227286 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:24.227289 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:24.228251 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:24.228261 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:24.228731 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:24.228735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:24.228914 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:24.229480 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:24.229497 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:24.231086 (+ 1589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:24.232717 (+ 1631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:24.232723 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:24.232741 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:24.233030 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:24.233292 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:24.233295 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:24.233297 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:24.233443 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:24.233830 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:24.233838 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:24.234205 (+ 367us) client_negotiation.cc:769] Sending connection context
0927 12:46:24.234387 (+ 182us) client_negotiation.cc:240] Negotiation successful
0927 12:46:24.234745 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":105,"threads_started":1}
I20260927 12:46:24.236775 3786 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40307
I20260927 12:46:24.237007 3786 heartbeater.cc:464] Registering TS with master...
I20260927 12:46:24.237504 3786 heartbeater.cc:511] Master 127.0.98.62:40307 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 12:46:24.238582 3461 ts_manager.cc:194] Registered new tserver with Master: 698359fb7f9c403cbc6b5cd8bc362562 (127.0.98.2:41845)
I20260927 12:46:24.239293 3461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43397
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 12:46:24.253190 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40307
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:46:24.364357 3791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:46:24.364678 3791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:46:24.364871 3791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:46:24.369486 3791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:46:24.369585 3791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:46:24.369686 3791 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:46:24.375373 3791 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:46:24.376533 3791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:46:24.377580 3791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:46:24.387544 3799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:24.387544 3797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:24.387538 3796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:24.387802 3791 server_base.cc:1034] running on GCE node
I20260927 12:46:24.388190 3791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:46:24.388918 3791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:46:24.390100 3791 hybrid_clock.cc:648] HybridClock initialized: now 1790513184390060 us; error 50 us; skew 500 ppm
Sep 27 12:46:24 dist-test-slave-lkn4 krb5kdc[3416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513184, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:46:24.392935 3791 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 12:46:24.393834 3791 webserver.cc:460] Webserver started at http://127.0.98.3:40067/ using document root <none> and password file <none>
I20260927 12:46:24.394435 3791 fs_manager.cc:359] Metadata directory not provided
I20260927 12:46:24.394498 3791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:46:24.394670 3791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:46:24.396709 3791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-2/data/instance:
uuid: "806caeb29f044fe7a1211e456d323d0d"
format_stamp: "Formatted at 2026-09-27 12:46:24 on dist-test-slave-lkn4"
server_key: "2391216781905023411f28c80c1c72c4"
server_key_iv: "cad98fa0b409e6e69f8cb9d41179330a"
server_key_version: "encryptionkey@0"
I20260927 12:46:24.397241 3791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-2/wal/instance:
uuid: "806caeb29f044fe7a1211e456d323d0d"
format_stamp: "Formatted at 2026-09-27 12:46:24 on dist-test-slave-lkn4"
server_key: "2391216781905023411f28c80c1c72c4"
server_key_iv: "cad98fa0b409e6e69f8cb9d41179330a"
server_key_version: "encryptionkey@0"
I20260927 12:46:24.401103 3791 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 12:46:24.403786 3806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:24.405104 3791 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 12:46:24.405236 3791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-2/wal
uuid: "806caeb29f044fe7a1211e456d323d0d"
format_stamp: "Formatted at 2026-09-27 12:46:24 on dist-test-slave-lkn4"
server_key: "2391216781905023411f28c80c1c72c4"
server_key_iv: "cad98fa0b409e6e69f8cb9d41179330a"
server_key_version: "encryptionkey@0"
I20260927 12:46:24.405359 3791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:46:24.433466 3791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:46:24.436800 3791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:46:24.436987 3791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:46:24.437633 3791 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:46:24.438506 3791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:46:24.438552 3791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:24.438596 3791 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:46:24.438623 3791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:24.447484 3791 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45297
I20260927 12:46:24.447510 3919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45297 every 8 connection(s)
I20260927 12:46:24.448423 3791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:46:24.449564 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 3791
I20260927 12:46:24.449666 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790513161160093-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:46:24.449982 392 external_mini_cluster.cc:1428] Setting key 09bb0b4dabba7a096b3502e2263658ee
Sep 27 12:46:24 dist-test-slave-lkn4 krb5kdc[3416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513184, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:46:24.463708 3500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:24.450522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40273 (local address 127.0.98.62:40307)
0927 12:46:24.450661 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:24.450665 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:24.451761 (+ 1096us) server_negotiation.cc:407] Connection header received
0927 12:46:24.453391 (+ 1630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:24.453394 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:24.453442 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:24.453503 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:24.454953 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:24.455511 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:24.456885 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:24.457206 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:24.457296 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:24.459949 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:24.459953 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:24.459955 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:24.459981 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:24.461338 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:24.461927 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:24.461930 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:24.461931 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:24.462008 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:24.462369 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:24.462371 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:24.462372 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:24.462556 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:24.462632 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:24.463418 (+ 786us) server_negotiation.cc:299] Negotiation successful
0927 12:46:24.463544 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 12:46:24.464615 3922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:24.450973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40307 (local address 127.0.98.3:40273)
0927 12:46:24.451555 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:24.451670 (+ 115us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:24.453173 (+ 1503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:24.453740 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:24.453748 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:24.454181 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:24.454754 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:24.454771 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:24.455717 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:24.455720 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:24.456727 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:24.456764 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:24.457374 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:24.457377 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:24.457546 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:24.458189 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:24.458209 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:24.459786 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:24.461445 (+ 1659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:24.461450 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:24.461465 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:24.461831 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:24.462110 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:24.462112 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:24.462114 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:24.462287 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:24.462689 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:24.462696 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:24.463004 (+ 308us) client_negotiation.cc:769] Sending connection context
0927 12:46:24.463221 (+ 217us) client_negotiation.cc:240] Negotiation successful
0927 12:46:24.463572 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":136,"threads_started":1}
I20260927 12:46:24.465734 3920 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40307
I20260927 12:46:24.466085 3920 heartbeater.cc:464] Registering TS with master...
I20260927 12:46:24.466590 3920 heartbeater.cc:511] Master 127.0.98.62:40307 requested a full tablet report, sending...
I20260927 12:46:24.467686 3461 ts_manager.cc:194] Registered new tserver with Master: 806caeb29f044fe7a1211e456d323d0d (127.0.98.3:45297)
I20260927 12:46:24.468251 3461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40273
I20260927 12:46:24.476676 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 12:46:24 dist-test-slave-lkn4 krb5kdc[3416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513184, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 27 12:46:24 dist-test-slave-lkn4 krb5kdc[3416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513184, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:46:24.501870 3500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:24.493521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44350 (local address 127.0.98.62:40307)
0927 12:46:24.493646 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:24.493651 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:24.493887 (+ 236us) server_negotiation.cc:407] Connection header received
0927 12:46:24.494021 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:24.494024 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:24.494106 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:24.494175 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:24.494904 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:24.495437 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:24.496276 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:24.496491 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:24.496576 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:24.499041 (+ 2465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:24.499044 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:24.499046 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:24.499072 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:24.500316 (+ 1244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:24.500775 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:24.500778 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:24.500780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:24.500863 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:24.501066 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:24.501068 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:24.501070 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:24.501296 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:24.501394 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:24.501605 (+ 211us) server_negotiation.cc:299] Negotiation successful
0927 12:46:24.501704 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260927 12:46:24.502391 3461 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:44350
I20260927 12:46:24.503405 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 3523
I20260927 12:46:24.512003 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 3657
I20260927 12:46:24.517125 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 3791
I20260927 12:46:24.522379 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 3429
2026-09-27T12:46:24Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2081 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:46:24 dist-test-slave-lkn4 krb5kdc[3932](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:46:24 dist-test-slave-lkn4 krb5kdc[3932](info): set up 2 sockets
Sep 27 12:46:24 dist-test-slave-lkn4 krb5kdc[3932](info): commencing operation
krb5kdc: starting...
W20260927 12:46:25.493911 392 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/krb5kdc/test-user.keytab.
Sep 27 12:46:25 dist-test-slave-lkn4 krb5kdc[3932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:46:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:46:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:46:25.614473 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36885
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36885
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:46:25.724557 3945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:46:25.724960 3945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:46:25.725008 3945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:46:25.729120 3945 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:46:25.729192 3945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:46:25.729210 3945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:46:25.729226 3945 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:46:25.729240 3945 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:46:25.734730 3945 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:46:25.735872 3945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:46:25.736920 3945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:46:25.745559 3953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:25.745555 3951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:25.745574 3950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:25.745782 3945 server_base.cc:1034] running on GCE node
I20260927 12:46:25.746631 3945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:46:25.747809 3945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:46:25.748972 3945 hybrid_clock.cc:648] HybridClock initialized: now 1790513185748953 us; error 33 us; skew 500 ppm
Sep 27 12:46:25 dist-test-slave-lkn4 krb5kdc[3932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513185, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:46:25.751802 3945 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:46:25.752791 3945 webserver.cc:460] Webserver started at http://127.0.98.62:43791/ using document root <none> and password file <none>
I20260927 12:46:25.753331 3945 fs_manager.cc:359] Metadata directory not provided
I20260927 12:46:25.753396 3945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:46:25.753587 3945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:46:25.755669 3945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/minicluster-data/master-0/data/instance:
uuid: "fafef9c4d62844a5a591b926a107ac4a"
format_stamp: "Formatted at 2026-09-27 12:46:25 on dist-test-slave-lkn4"
server_key: "62a4d15096552fcf615f9e14e18cb8d5"
server_key_iv: "1e0557989ae3fabd8da68bf3ae66bf38"
server_key_version: "encryptionkey@0"
I20260927 12:46:25.756219 3945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/minicluster-data/master-0/wal/instance:
uuid: "fafef9c4d62844a5a591b926a107ac4a"
format_stamp: "Formatted at 2026-09-27 12:46:25 on dist-test-slave-lkn4"
server_key: "62a4d15096552fcf615f9e14e18cb8d5"
server_key_iv: "1e0557989ae3fabd8da68bf3ae66bf38"
server_key_version: "encryptionkey@0"
I20260927 12:46:25.760043 3945 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 12:46:25.762704 3960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:25.764097 3945 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:46:25.764225 3945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/minicluster-data/master-0/wal
uuid: "fafef9c4d62844a5a591b926a107ac4a"
format_stamp: "Formatted at 2026-09-27 12:46:25 on dist-test-slave-lkn4"
server_key: "62a4d15096552fcf615f9e14e18cb8d5"
server_key_iv: "1e0557989ae3fabd8da68bf3ae66bf38"
server_key_version: "encryptionkey@0"
I20260927 12:46:25.764324 3945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:46:25.780719 3945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:46:25.783962 3945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:46:25.784157 3945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:46:25.790763 3945 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36885
I20260927 12:46:25.790880 4012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36885 every 8 connection(s)
I20260927 12:46:25.791739 3945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:46:25.794579 4013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:46:25.799990 4013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fafef9c4d62844a5a591b926a107ac4a: Bootstrap starting.
I20260927 12:46:25.800998 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 3945
I20260927 12:46:25.801133 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/minicluster-data/master-0/wal/instance
I20260927 12:46:25.801433 392 external_mini_cluster.cc:1428] Setting key 488efb7abc7f05e54b75b43ecba692ff
I20260927 12:46:25.802858 4013 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fafef9c4d62844a5a591b926a107ac4a: Neither blocks nor log segments found. Creating new log.
I20260927 12:46:25.803565 4013 log.cc:824] T 00000000000000000000000000000000 P fafef9c4d62844a5a591b926a107ac4a: Log is configured to *not* fsync() on all Append() calls
I20260927 12:46:25.805645 4013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fafef9c4d62844a5a591b926a107ac4a: No bootstrap required, opened a new log
I20260927 12:46:25.809803 4013 raft_consensus.cc:348] T 00000000000000000000000000000000 P fafef9c4d62844a5a591b926a107ac4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fafef9c4d62844a5a591b926a107ac4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36885 } }
I20260927 12:46:25.810029 4013 raft_consensus.cc:374] T 00000000000000000000000000000000 P fafef9c4d62844a5a591b926a107ac4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:46:25.810108 4013 raft_consensus.cc:729] T 00000000000000000000000000000000 P fafef9c4d62844a5a591b926a107ac4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fafef9c4d62844a5a591b926a107ac4a, State: Initialized, Role: FOLLOWER
Sep 27 12:46:25 dist-test-slave-lkn4 krb5kdc[3932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:46:25.810560 4013 consensus_queue.cc:260] T 00000000000000000000000000000000 P fafef9c4d62844a5a591b926a107ac4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fafef9c4d62844a5a591b926a107ac4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36885 } }
I20260927 12:46:25.810686 4013 raft_consensus.cc:388] T 00000000000000000000000000000000 P fafef9c4d62844a5a591b926a107ac4a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:46:25.810765 4013 raft_consensus.cc:482] T 00000000000000000000000000000000 P fafef9c4d62844a5a591b926a107ac4a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:46:25.810864 4013 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fafef9c4d62844a5a591b926a107ac4a [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:46:25.811848 4013 raft_consensus.cc:504] T 00000000000000000000000000000000 P fafef9c4d62844a5a591b926a107ac4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fafef9c4d62844a5a591b926a107ac4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36885 } }
I20260927 12:46:25.812199 4013 leader_election.cc:302] T 00000000000000000000000000000000 P fafef9c4d62844a5a591b926a107ac4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fafef9c4d62844a5a591b926a107ac4a; no voters:
I20260927 12:46:25.812503 4013 leader_election.cc:288] T 00000000000000000000000000000000 P fafef9c4d62844a5a591b926a107ac4a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:46:25.812621 4018 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fafef9c4d62844a5a591b926a107ac4a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:46:25.812829 4018 raft_consensus.cc:686] T 00000000000000000000000000000000 P fafef9c4d62844a5a591b926a107ac4a [term 1 LEADER]: Becoming Leader. State: Replica: fafef9c4d62844a5a591b926a107ac4a, State: Running, Role: LEADER
I20260927 12:46:25.813215 4018 consensus_queue.cc:237] T 00000000000000000000000000000000 P fafef9c4d62844a5a591b926a107ac4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fafef9c4d62844a5a591b926a107ac4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36885 } }
I20260927 12:46:25.813838 4013 sys_catalog.cc:564] T 00000000000000000000000000000000 P fafef9c4d62844a5a591b926a107ac4a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:46:25.814898 4020 sys_catalog.cc:455] T 00000000000000000000000000000000 P fafef9c4d62844a5a591b926a107ac4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader fafef9c4d62844a5a591b926a107ac4a. Latest consensus state: current_term: 1 leader_uuid: "fafef9c4d62844a5a591b926a107ac4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fafef9c4d62844a5a591b926a107ac4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36885 } } }
I20260927 12:46:25.815027 4020 sys_catalog.cc:458] T 00000000000000000000000000000000 P fafef9c4d62844a5a591b926a107ac4a [sys.catalog]: This master's current role is: LEADER
I20260927 12:46:25.815352 4019 sys_catalog.cc:455] T 00000000000000000000000000000000 P fafef9c4d62844a5a591b926a107ac4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fafef9c4d62844a5a591b926a107ac4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fafef9c4d62844a5a591b926a107ac4a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36885 } } }
I20260927 12:46:25.815428 4027 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:46:25.815440 4019 sys_catalog.cc:458] T 00000000000000000000000000000000 P fafef9c4d62844a5a591b926a107ac4a [sys.catalog]: This master's current role is: LEADER
I20260927 12:46:25.816596 4016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:25.802564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60540 (local address 127.0.98.62:36885)
0927 12:46:25.802987 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:25.803006 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:25.803051 (+ 45us) server_negotiation.cc:407] Connection header received
0927 12:46:25.803663 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:25.803685 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:25.804049 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:25.804332 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:25.805152 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:25.806227 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:25.807390 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:25.807961 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:25.808087 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:25.810834 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:25.810840 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:25.810859 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:25.810900 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:25.812809 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:25.813390 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:25.813396 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:25.813402 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:25.813535 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:25.813868 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:25.813872 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:25.813873 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:25.814418 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:25.814605 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:25.815252 (+ 647us) server_negotiation.cc:299] Negotiation successful
0927 12:46:25.815590 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":116,"threads_started":1}
I20260927 12:46:25.818655 4027 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:46:25.824580 4027 catalog_manager.cc:1347] Generated new cluster ID: 279856057e794831abadd13eb0f55593
I20260927 12:46:25.824658 4027 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:46:25.845774 4027 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:46:25.847219 4027 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:46:25.860208 4027 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fafef9c4d62844a5a591b926a107ac4a: Generated new TSK 0
I20260927 12:46:25.861091 4027 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 12:46:25.889914 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36885
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:46:26.020473 4039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:46:26.020771 4039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:46:26.020816 4039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:46:26.025043 4039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:46:26.025126 4039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:46:26.025202 4039 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:46:26.030799 4039 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:46:26.031911 4039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:46:26.032974 4039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 12:46:26.043126 4039 server_base.cc:1034] running on GCE node
W20260927 12:46:26.043092 4044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:26.043087 4047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:26.043087 4045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:26.043782 4039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:46:26.044503 4039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:46:26.047814 4039 hybrid_clock.cc:648] HybridClock initialized: now 1790513186047743 us; error 82 us; skew 500 ppm
Sep 27 12:46:26 dist-test-slave-lkn4 krb5kdc[3932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513186, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:46:26.050657 4039 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 12:46:26.051604 4039 webserver.cc:460] Webserver started at http://127.0.98.1:36233/ using document root <none> and password file <none>
I20260927 12:46:26.052160 4039 fs_manager.cc:359] Metadata directory not provided
I20260927 12:46:26.052225 4039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:46:26.052448 4039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:46:26.055032 4039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/minicluster-data/ts-0/data/instance:
uuid: "81b2af129cd64332a0c009b0d4872782"
format_stamp: "Formatted at 2026-09-27 12:46:26 on dist-test-slave-lkn4"
server_key: "57b8afb3df0e18ade7b65d7359fa2f23"
server_key_iv: "64f34bfa42f8a1cb0e9bc30f1e3361b9"
server_key_version: "encryptionkey@0"
I20260927 12:46:26.055657 4039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/minicluster-data/ts-0/wal/instance:
uuid: "81b2af129cd64332a0c009b0d4872782"
format_stamp: "Formatted at 2026-09-27 12:46:26 on dist-test-slave-lkn4"
server_key: "57b8afb3df0e18ade7b65d7359fa2f23"
server_key_iv: "64f34bfa42f8a1cb0e9bc30f1e3361b9"
server_key_version: "encryptionkey@0"
I20260927 12:46:26.059719 4039 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 12:46:26.062690 4054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:26.064078 4039 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:46:26.064204 4039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/minicluster-data/ts-0/wal
uuid: "81b2af129cd64332a0c009b0d4872782"
format_stamp: "Formatted at 2026-09-27 12:46:26 on dist-test-slave-lkn4"
server_key: "57b8afb3df0e18ade7b65d7359fa2f23"
server_key_iv: "64f34bfa42f8a1cb0e9bc30f1e3361b9"
server_key_version: "encryptionkey@0"
I20260927 12:46:26.064316 4039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:46:26.087240 4039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:46:26.091070 4039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:46:26.091276 4039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:46:26.091925 4039 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:46:26.092861 4039 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:46:26.092912 4039 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:26.092967 4039 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:46:26.092996 4039 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:26.101953 4039 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33691
I20260927 12:46:26.101974 4167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33691 every 8 connection(s)
I20260927 12:46:26.102957 4039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:46:26.106817 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 4039
I20260927 12:46:26.106930 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:46:26.107187 392 external_mini_cluster.cc:1428] Setting key 7d928599f5243287cd9c775973d00509
Sep 27 12:46:26 dist-test-slave-lkn4 krb5kdc[3932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513186, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 12:46:26.119225 4016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:26.104853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34319 (local address 127.0.98.62:36885)
0927 12:46:26.104978 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:26.104982 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:26.106098 (+ 1116us) server_negotiation.cc:407] Connection header received
0927 12:46:26.107649 (+ 1551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:26.107652 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:26.107710 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:26.107782 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:26.109135 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:26.109874 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:26.111545 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:26.111778 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:26.111866 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:26.115054 (+ 3188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:26.115058 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:26.115061 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:26.115089 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:26.116894 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:26.117481 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:26.117484 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:26.117485 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:26.117561 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:26.117903 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:26.117907 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:26.117909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:26.118153 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:26.118242 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:26.118849 (+ 607us) server_negotiation.cc:299] Negotiation successful
0927 12:46:26.119050 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 12:46:26.120191 4170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:26.105307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36885 (local address 127.0.98.1:34319)
0927 12:46:26.105917 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:26.105963 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:26.107437 (+ 1474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:26.107990 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:26.107998 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:26.108443 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:26.108973 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:26.108987 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:26.110019 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:26.110024 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:26.111411 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:26.111422 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:26.111949 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:26.111953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:26.112141 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:26.112941 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:26.112962 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:26.114866 (+ 1904us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:26.117024 (+ 2158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:26.117031 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:26.117051 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:26.117368 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:26.117646 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:26.117649 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:26.117652 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:26.117806 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:26.118259 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:26.118267 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:26.118590 (+ 323us) client_negotiation.cc:769] Sending connection context
0927 12:46:26.118776 (+ 186us) client_negotiation.cc:240] Negotiation successful
0927 12:46:26.119156 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":111,"threads_started":1}
I20260927 12:46:26.121563 4168 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36885
I20260927 12:46:26.121819 4168 heartbeater.cc:464] Registering TS with master...
I20260927 12:46:26.122426 4168 heartbeater.cc:511] Master 127.0.98.62:36885 requested a full tablet report, sending...
I20260927 12:46:26.124084 3977 ts_manager.cc:194] Registered new tserver with Master: 81b2af129cd64332a0c009b0d4872782 (127.0.98.1:33691)
I20260927 12:46:26.125236 3977 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34319
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 12:46:26.137516 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36885
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:46:26.248797 4173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:46:26.249065 4173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:46:26.249110 4173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:46:26.253384 4173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:46:26.253453 4173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:46:26.253518 4173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:46:26.259392 4173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:46:26.260502 4173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:46:26.261546 4173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:46:26.271152 4181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:26.271188 4178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:26.271170 4179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:26.271477 4173 server_base.cc:1034] running on GCE node
I20260927 12:46:26.271898 4173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:46:26.272559 4173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:46:26.273733 4173 hybrid_clock.cc:648] HybridClock initialized: now 1790513186273725 us; error 73 us; skew 500 ppm
Sep 27 12:46:26 dist-test-slave-lkn4 krb5kdc[3932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513186, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:46:26.276530 4173 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 12:46:26.277462 4173 webserver.cc:460] Webserver started at http://127.0.98.2:35499/ using document root <none> and password file <none>
I20260927 12:46:26.277994 4173 fs_manager.cc:359] Metadata directory not provided
I20260927 12:46:26.278088 4173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:46:26.278334 4173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:46:26.280565 4173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/minicluster-data/ts-1/data/instance:
uuid: "24b1dd0e442e470389a2ca3a90faf046"
format_stamp: "Formatted at 2026-09-27 12:46:26 on dist-test-slave-lkn4"
server_key: "9242b5b46e0da37868c8508811fdd15f"
server_key_iv: "33a69d33d553a939a097b89aa29fed45"
server_key_version: "encryptionkey@0"
I20260927 12:46:26.281134 4173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/minicluster-data/ts-1/wal/instance:
uuid: "24b1dd0e442e470389a2ca3a90faf046"
format_stamp: "Formatted at 2026-09-27 12:46:26 on dist-test-slave-lkn4"
server_key: "9242b5b46e0da37868c8508811fdd15f"
server_key_iv: "33a69d33d553a939a097b89aa29fed45"
server_key_version: "encryptionkey@0"
I20260927 12:46:26.285010 4173 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 12:46:26.287569 4188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:26.288843 4173 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 12:46:26.288971 4173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/minicluster-data/ts-1/wal
uuid: "24b1dd0e442e470389a2ca3a90faf046"
format_stamp: "Formatted at 2026-09-27 12:46:26 on dist-test-slave-lkn4"
server_key: "9242b5b46e0da37868c8508811fdd15f"
server_key_iv: "33a69d33d553a939a097b89aa29fed45"
server_key_version: "encryptionkey@0"
I20260927 12:46:26.289074 4173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:46:26.313606 4173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:46:26.317292 4173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:46:26.317469 4173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:46:26.318106 4173 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:46:26.318967 4173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:46:26.319020 4173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:26.319067 4173 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:46:26.319087 4173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:26.327327 4173 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35785
I20260927 12:46:26.327353 4301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35785 every 8 connection(s)
I20260927 12:46:26.328243 4173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:46:26.334272 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 4173
I20260927 12:46:26.334363 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:46:26.334609 392 external_mini_cluster.cc:1428] Setting key b8689f9e4427895242e27aa23bd7fb75
Sep 27 12:46:26 dist-test-slave-lkn4 krb5kdc[3932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513186, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:46:26.342949 4016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:26.330235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54077 (local address 127.0.98.62:36885)
0927 12:46:26.330372 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:26.330376 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:26.331322 (+ 946us) server_negotiation.cc:407] Connection header received
0927 12:46:26.332729 (+ 1407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:26.332733 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:26.332792 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:26.332861 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:26.334141 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:26.334726 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:26.335970 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:26.336310 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:26.336430 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:26.339303 (+ 2873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:26.339307 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:26.339309 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:26.339332 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:26.340864 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:26.341389 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:26.341392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:26.341393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:26.341472 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:26.341804 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:26.341807 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:26.341809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:26.341993 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:26.342110 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:26.342645 (+ 535us) server_negotiation.cc:299] Negotiation successful
0927 12:46:26.342776 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 12:46:26.343878 4304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:26.330709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36885 (local address 127.0.98.2:54077)
0927 12:46:26.331166 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:26.331210 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:26.332513 (+ 1303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:26.333048 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:26.333055 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:26.333433 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:26.333944 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:26.333959 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:26.334845 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:26.334848 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:26.335841 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:26.335853 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:26.336464 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:26.336468 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:26.336624 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:26.337347 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:26.337364 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:26.339132 (+ 1768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:26.340984 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:26.340990 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:26.341007 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:26.341291 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:26.341554 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:26.341557 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:26.341559 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:26.341705 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:26.342116 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:26.342122 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:26.342407 (+ 285us) client_negotiation.cc:769] Sending connection context
0927 12:46:26.342587 (+ 180us) client_negotiation.cc:240] Negotiation successful
0927 12:46:26.342952 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":111,"threads_started":1}
I20260927 12:46:26.345237 4302 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36885
I20260927 12:46:26.345490 4302 heartbeater.cc:464] Registering TS with master...
I20260927 12:46:26.346060 4302 heartbeater.cc:511] Master 127.0.98.62:36885 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 12:46:26.347275 3977 ts_manager.cc:194] Registered new tserver with Master: 24b1dd0e442e470389a2ca3a90faf046 (127.0.98.2:35785)
I20260927 12:46:26.348055 3977 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54077
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 12:46:26.362112 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36885
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:46:26.472286 4307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:46:26.472599 4307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:46:26.472662 4307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:46:26.476836 4307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:46:26.476914 4307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:46:26.476998 4307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:46:26.482422 4307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:46:26.483539 4307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:46:26.484578 4307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:46:26.494402 4313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:26.494403 4312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:26.494684 4307 server_base.cc:1034] running on GCE node
W20260927 12:46:26.494627 4315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:26.495286 4307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:46:26.495957 4307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:46:26.497197 4307 hybrid_clock.cc:648] HybridClock initialized: now 1790513186497166 us; error 43 us; skew 500 ppm
Sep 27 12:46:26 dist-test-slave-lkn4 krb5kdc[3932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513186, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:46:26.500087 4307 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 12:46:26.501065 4307 webserver.cc:460] Webserver started at http://127.0.98.3:44085/ using document root <none> and password file <none>
I20260927 12:46:26.501616 4307 fs_manager.cc:359] Metadata directory not provided
I20260927 12:46:26.501678 4307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:46:26.501884 4307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:46:26.503952 4307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/minicluster-data/ts-2/data/instance:
uuid: "6ae3e3da47d642c2992d88a2a4cd1f0e"
format_stamp: "Formatted at 2026-09-27 12:46:26 on dist-test-slave-lkn4"
server_key: "3f0e553b33a46ab1ce0bfcfc808ab8d2"
server_key_iv: "6668e3c4ffebf9146c5219878e63409c"
server_key_version: "encryptionkey@0"
I20260927 12:46:26.504513 4307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6ae3e3da47d642c2992d88a2a4cd1f0e"
format_stamp: "Formatted at 2026-09-27 12:46:26 on dist-test-slave-lkn4"
server_key: "3f0e553b33a46ab1ce0bfcfc808ab8d2"
server_key_iv: "6668e3c4ffebf9146c5219878e63409c"
server_key_version: "encryptionkey@0"
I20260927 12:46:26.508580 4307 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 12:46:26.511188 4322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:26.512590 4307 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:46:26.512778 4307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/minicluster-data/ts-2/wal
uuid: "6ae3e3da47d642c2992d88a2a4cd1f0e"
format_stamp: "Formatted at 2026-09-27 12:46:26 on dist-test-slave-lkn4"
server_key: "3f0e553b33a46ab1ce0bfcfc808ab8d2"
server_key_iv: "6668e3c4ffebf9146c5219878e63409c"
server_key_version: "encryptionkey@0"
I20260927 12:46:26.512892 4307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:46:26.534785 4307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:46:26.538074 4307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:46:26.538290 4307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:46:26.538866 4307 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:46:26.539734 4307 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:46:26.539783 4307 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:26.539836 4307 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:46:26.539866 4307 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:26.548461 4307 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36351
I20260927 12:46:26.548491 4435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36351 every 8 connection(s)
I20260927 12:46:26.549345 4307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 12:46:26 dist-test-slave-lkn4 krb5kdc[3932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513186, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:46:26.558698 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 4307
I20260927 12:46:26.558907 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790513161160093-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:46:26.559242 392 external_mini_cluster.cc:1428] Setting key 15247f11198e409be421d6d6aaa092f8
I20260927 12:46:26.562846 4016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:26.551048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48623 (local address 127.0.98.62:36885)
0927 12:46:26.551167 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:26.551171 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:26.552042 (+ 871us) server_negotiation.cc:407] Connection header received
0927 12:46:26.553419 (+ 1377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:26.553423 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:26.553489 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:26.553565 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:26.554804 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:26.555335 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:26.556438 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:26.556661 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:26.556756 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:26.559207 (+ 2451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:26.559210 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:26.559212 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:26.559244 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:26.560725 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:26.561313 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:26.561316 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:26.561318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:26.561418 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:26.561815 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:26.561818 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:26.561819 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:26.562006 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:26.562153 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:26.562618 (+ 465us) server_negotiation.cc:299] Negotiation successful
0927 12:46:26.562730 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 12:46:26.563669 4438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:26.551446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36885 (local address 127.0.98.3:48623)
0927 12:46:26.551889 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:26.551934 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:26.553197 (+ 1263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:26.553776 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:26.553784 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:26.554179 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:26.554664 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:26.554677 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:26.555446 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:26.555449 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:26.556329 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:26.556337 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:26.556796 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:26.556801 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:26.556959 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:26.557469 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:26.557485 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:26.559024 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:26.560830 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:26.560836 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:26.560851 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:26.561189 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:26.561507 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:26.561510 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:26.561512 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:26.561661 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:26.562149 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:26.562154 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:26.562400 (+ 246us) client_negotiation.cc:769] Sending connection context
0927 12:46:26.562558 (+ 158us) client_negotiation.cc:240] Negotiation successful
0927 12:46:26.562821 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":107,"threads_started":1}
I20260927 12:46:26.564714 4436 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36885
I20260927 12:46:26.565011 4436 heartbeater.cc:464] Registering TS with master...
I20260927 12:46:26.565479 4436 heartbeater.cc:511] Master 127.0.98.62:36885 requested a full tablet report, sending...
I20260927 12:46:26.566504 3977 ts_manager.cc:194] Registered new tserver with Master: 6ae3e3da47d642c2992d88a2a4cd1f0e (127.0.98.3:36351)
I20260927 12:46:26.567095 3977 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48623
I20260927 12:46:26.573043 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 12:46:26 dist-test-slave-lkn4 krb5kdc[3932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 12:46:26 dist-test-slave-lkn4 krb5kdc[3932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 12:46:26.599275 4443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:26.589052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37188 (local address 127.0.98.1:33691)
0927 12:46:26.589354 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:26.589371 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:26.589398 (+ 27us) server_negotiation.cc:407] Connection header received
0927 12:46:26.589450 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:26.589456 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:26.589645 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:26.589785 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:26.590524 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:26.591359 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:26.592369 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:26.592780 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:26.592874 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:26.595327 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:26.595337 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:26.595352 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:26.595380 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:26.597161 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:26.597571 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:26.597576 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:26.597578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:26.597727 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:26.598027 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:26.598030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:26.598032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:26.598368 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:26.598562 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:26.598888 (+ 326us) server_negotiation.cc:299] Negotiation successful
0927 12:46:26.599073 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":78,"threads_started":1}
Sep 27 12:46:26 dist-test-slave-lkn4 krb5kdc[3932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:46:26.610877 4016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:26.602463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60554 (local address 127.0.98.62:36885)
0927 12:46:26.602617 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:26.602623 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:26.602662 (+ 39us) server_negotiation.cc:407] Connection header received
0927 12:46:26.602808 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:26.602811 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:26.602857 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:26.602923 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:26.603551 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:26.604109 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:26.604898 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:26.605118 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:26.605207 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:26.607870 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:26.607874 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:26.607876 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:26.607903 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:26.609138 (+ 1235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:26.609557 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:26.609562 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:26.609564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:26.609642 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:26.609923 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:26.609926 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:26.609928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:26.610222 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:26.610366 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:26.610591 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 12:46:26.610723 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260927 12:46:26.611340 3977 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60554
I20260927 12:46:26.612114 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 4039
I20260927 12:46:26.617667 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 4173
I20260927 12:46:26.622182 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 4307
I20260927 12:46:26.626819 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 3945
2026-09-27T12:46:26Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2102 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:46:26 dist-test-slave-lkn4 krb5kdc[4448](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:46:26 dist-test-slave-lkn4 krb5kdc[4448](info): set up 2 sockets
Sep 27 12:46:26 dist-test-slave-lkn4 krb5kdc[4448](info): commencing operation
krb5kdc: starting...
W20260927 12:46:27.586022 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/krb5kdc/test-user.keytab.
Sep 27 12:46:27 dist-test-slave-lkn4 krb5kdc[4448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:46:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:46:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:46:27.705338 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36697
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36697
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:46:27.815673 4461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:46:27.816176 4461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:46:27.816372 4461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:46:27.821452 4461 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:46:27.821538 4461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:46:27.821558 4461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:46:27.821574 4461 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:46:27.821589 4461 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:46:27.827287 4461 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36697
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:46:27.828321 4461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:46:27.829305 4461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:46:27.838490 4469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:27.838519 4466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:27.838521 4467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:27.838757 4461 server_base.cc:1034] running on GCE node
I20260927 12:46:27.839342 4461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:46:27.840631 4461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:46:27.841833 4461 hybrid_clock.cc:648] HybridClock initialized: now 1790513187841806 us; error 42 us; skew 500 ppm
Sep 27 12:46:27 dist-test-slave-lkn4 krb5kdc[4448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513187, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:46:27.844687 4461 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:46:27.845407 4461 fs_manager.cc:359] Metadata directory not provided
I20260927 12:46:27.845470 4461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:46:27.845638 4461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:46:27.847784 4461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/minicluster-data/master-0/data/instance:
uuid: "f2d0840784584dd1a6361c075eda2075"
format_stamp: "Formatted at 2026-09-27 12:46:27 on dist-test-slave-lkn4"
server_key: "78cfc52e85bae566c918ad987cf09b64"
server_key_iv: "7601f4460b292093d80816866a87fbb4"
server_key_version: "encryptionkey@0"
I20260927 12:46:27.848297 4461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/minicluster-data/master-0/wal/instance:
uuid: "f2d0840784584dd1a6361c075eda2075"
format_stamp: "Formatted at 2026-09-27 12:46:27 on dist-test-slave-lkn4"
server_key: "78cfc52e85bae566c918ad987cf09b64"
server_key_iv: "7601f4460b292093d80816866a87fbb4"
server_key_version: "encryptionkey@0"
I20260927 12:46:27.852232 4461 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 12:46:27.854817 4475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:27.856130 4461 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 12:46:27.856248 4461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/minicluster-data/master-0/wal
uuid: "f2d0840784584dd1a6361c075eda2075"
format_stamp: "Formatted at 2026-09-27 12:46:27 on dist-test-slave-lkn4"
server_key: "78cfc52e85bae566c918ad987cf09b64"
server_key_iv: "7601f4460b292093d80816866a87fbb4"
server_key_version: "encryptionkey@0"
I20260927 12:46:27.856365 4461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:46:27.877790 4461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:46:27.881058 4461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:46:27.881245 4461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:46:27.887940 4461 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36697
I20260927 12:46:27.888190 4527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36697 every 8 connection(s)
I20260927 12:46:27.888605 4461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:46:27.891327 4528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:46:27.891629 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 4461
I20260927 12:46:27.891793 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/minicluster-data/master-0/wal/instance
I20260927 12:46:27.892045 392 external_mini_cluster.cc:1428] Setting key 52e5ef04af90cf4ce33287b256dab14e
I20260927 12:46:27.898133 4528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2d0840784584dd1a6361c075eda2075: Bootstrap starting.
I20260927 12:46:27.900691 4528 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2d0840784584dd1a6361c075eda2075: Neither blocks nor log segments found. Creating new log.
Sep 27 12:46:27 dist-test-slave-lkn4 krb5kdc[4448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:46:27.901403 4528 log.cc:824] T 00000000000000000000000000000000 P f2d0840784584dd1a6361c075eda2075: Log is configured to *not* fsync() on all Append() calls
I20260927 12:46:27.903323 4528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2d0840784584dd1a6361c075eda2075: No bootstrap required, opened a new log
I20260927 12:46:27.905802 4531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:27.893303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32922 (local address 127.0.98.62:36697)
0927 12:46:27.893669 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:27.893685 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:27.893721 (+ 36us) server_negotiation.cc:407] Connection header received
0927 12:46:27.894496 (+ 775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:27.894517 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:27.894863 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:27.895138 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:27.895996 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:27.896989 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:27.898147 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:27.898687 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:27.898832 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:27.901384 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:27.901389 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:27.901402 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:27.901431 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:27.903216 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:27.903655 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:27.903660 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:27.903664 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:27.903778 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:27.904018 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:27.904021 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:27.904023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:27.904388 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:27.904564 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:27.904833 (+ 269us) server_negotiation.cc:299] Negotiation successful
0927 12:46:27.905096 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":97,"threads_started":1}
I20260927 12:46:27.907074 4528 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2d0840784584dd1a6361c075eda2075 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2d0840784584dd1a6361c075eda2075" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36697 } }
I20260927 12:46:27.907298 4528 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2d0840784584dd1a6361c075eda2075 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:46:27.907351 4528 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2d0840784584dd1a6361c075eda2075 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2d0840784584dd1a6361c075eda2075, State: Initialized, Role: FOLLOWER
I20260927 12:46:27.907775 4528 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2d0840784584dd1a6361c075eda2075 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2d0840784584dd1a6361c075eda2075" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36697 } }
I20260927 12:46:27.907915 4528 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2d0840784584dd1a6361c075eda2075 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:46:27.908006 4528 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2d0840784584dd1a6361c075eda2075 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:46:27.908099 4528 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2d0840784584dd1a6361c075eda2075 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:46:27.909018 4528 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2d0840784584dd1a6361c075eda2075 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2d0840784584dd1a6361c075eda2075" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36697 } }
I20260927 12:46:27.909332 4528 leader_election.cc:302] T 00000000000000000000000000000000 P f2d0840784584dd1a6361c075eda2075 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2d0840784584dd1a6361c075eda2075; no voters:
I20260927 12:46:27.909605 4528 leader_election.cc:288] T 00000000000000000000000000000000 P f2d0840784584dd1a6361c075eda2075 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:46:27.909727 4533 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2d0840784584dd1a6361c075eda2075 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:46:27.909983 4533 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2d0840784584dd1a6361c075eda2075 [term 1 LEADER]: Becoming Leader. State: Replica: f2d0840784584dd1a6361c075eda2075, State: Running, Role: LEADER
I20260927 12:46:27.910320 4533 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2d0840784584dd1a6361c075eda2075 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2d0840784584dd1a6361c075eda2075" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36697 } }
I20260927 12:46:27.910799 4528 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2d0840784584dd1a6361c075eda2075 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:46:27.911803 4534 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2d0840784584dd1a6361c075eda2075 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2d0840784584dd1a6361c075eda2075" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2d0840784584dd1a6361c075eda2075" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36697 } } }
I20260927 12:46:27.911927 4534 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2d0840784584dd1a6361c075eda2075 [sys.catalog]: This master's current role is: LEADER
I20260927 12:46:27.912186 4535 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2d0840784584dd1a6361c075eda2075 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2d0840784584dd1a6361c075eda2075. Latest consensus state: current_term: 1 leader_uuid: "f2d0840784584dd1a6361c075eda2075" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2d0840784584dd1a6361c075eda2075" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36697 } } }
I20260927 12:46:27.912309 4535 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2d0840784584dd1a6361c075eda2075 [sys.catalog]: This master's current role is: LEADER
I20260927 12:46:27.912391 4542 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:46:27.915613 4542 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:46:27.921667 4542 catalog_manager.cc:1347] Generated new cluster ID: a781ff089973467180345735dd586eb1
I20260927 12:46:27.921751 4542 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:46:27.934224 4542 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:46:27.935242 4542 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:46:27.946871 4542 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2d0840784584dd1a6361c075eda2075: Generated new TSK 0
I20260927 12:46:27.947645 4542 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 12:46:27.982259 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36697
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:46:28.117651 4554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:46:28.117964 4554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:46:28.118012 4554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:46:28.122146 4554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:46:28.122225 4554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:46:28.122288 4554 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:46:28.127813 4554 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:46:28.128880 4554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:46:28.129870 4554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:46:28.139380 4560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:28.139381 4562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:28.139381 4559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:28.139724 4554 server_base.cc:1034] running on GCE node
I20260927 12:46:28.140180 4554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:46:28.140884 4554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:46:28.142119 4554 hybrid_clock.cc:648] HybridClock initialized: now 1790513188142062 us; error 57 us; skew 500 ppm
Sep 27 12:46:28 dist-test-slave-lkn4 krb5kdc[4448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513188, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:46:28.145076 4554 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 12:46:28.145859 4554 fs_manager.cc:359] Metadata directory not provided
I20260927 12:46:28.145932 4554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:46:28.146185 4554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:46:28.148319 4554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/minicluster-data/ts-0/data/instance:
uuid: "946836d1a14e43ce82c08e8b6bcf41cd"
format_stamp: "Formatted at 2026-09-27 12:46:28 on dist-test-slave-lkn4"
server_key: "4ea2b6f7b34b2ee5aedd3ec075f8787e"
server_key_iv: "f87bfe1bcdf8dcf5d3009a3877c96b75"
server_key_version: "encryptionkey@0"
I20260927 12:46:28.148900 4554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/minicluster-data/ts-0/wal/instance:
uuid: "946836d1a14e43ce82c08e8b6bcf41cd"
format_stamp: "Formatted at 2026-09-27 12:46:28 on dist-test-slave-lkn4"
server_key: "4ea2b6f7b34b2ee5aedd3ec075f8787e"
server_key_iv: "f87bfe1bcdf8dcf5d3009a3877c96b75"
server_key_version: "encryptionkey@0"
I20260927 12:46:28.152747 4554 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260927 12:46:28.155315 4568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:28.156615 4554 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 12:46:28.156734 4554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/minicluster-data/ts-0/wal
uuid: "946836d1a14e43ce82c08e8b6bcf41cd"
format_stamp: "Formatted at 2026-09-27 12:46:28 on dist-test-slave-lkn4"
server_key: "4ea2b6f7b34b2ee5aedd3ec075f8787e"
server_key_iv: "f87bfe1bcdf8dcf5d3009a3877c96b75"
server_key_version: "encryptionkey@0"
I20260927 12:46:28.156840 4554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:46:28.177510 4554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:46:28.181336 4554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:46:28.181520 4554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:46:28.182163 4554 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:46:28.183056 4554 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:46:28.183103 4554 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:28.183157 4554 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:46:28.183185 4554 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:28.191601 4554 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35695
I20260927 12:46:28.191679 4681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35695 every 8 connection(s)
I20260927 12:46:28.192196 4554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:46:28.200621 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 4554
I20260927 12:46:28.200734 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:46:28.200989 392 external_mini_cluster.cc:1428] Setting key 64889cdd996104cf84f714ea5fd25254
Sep 27 12:46:28 dist-test-slave-lkn4 krb5kdc[4448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513188, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:46:28.205355 4531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:28.193876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60457 (local address 127.0.98.62:36697)
0927 12:46:28.194019 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:28.194023 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:28.194950 (+ 927us) server_negotiation.cc:407] Connection header received
0927 12:46:28.196270 (+ 1320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:28.196273 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:28.196327 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:28.196401 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:28.197590 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:28.198188 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:28.199339 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:28.199565 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:28.199644 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:28.201996 (+ 2352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:28.202000 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:28.202001 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:28.202025 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:28.203387 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:28.203881 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:28.203883 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:28.203884 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:28.203961 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:28.204275 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:28.204279 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:28.204281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:28.204455 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:28.204539 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:28.205046 (+ 507us) server_negotiation.cc:299] Negotiation successful
0927 12:46:28.205175 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 12:46:28.206357 4684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:28.194331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36697 (local address 127.0.98.1:60457)
0927 12:46:28.194797 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:28.194841 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:28.196038 (+ 1197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:28.196555 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:28.196562 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:28.196928 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:28.197453 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:28.197468 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:28.198326 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:28.198329 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:28.199212 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:28.199221 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:28.199685 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:28.199687 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:28.199824 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:28.200376 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:28.200393 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:28.201834 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:28.203497 (+ 1663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:28.203503 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:28.203522 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:28.203787 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:28.204038 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:28.204042 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:28.204044 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:28.204185 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:28.204550 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:28.204557 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:28.204815 (+ 258us) client_negotiation.cc:769] Sending connection context
0927 12:46:28.204991 (+ 176us) client_negotiation.cc:240] Negotiation successful
0927 12:46:28.205328 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":112,"threads_started":1}
I20260927 12:46:28.207792 4682 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36697
I20260927 12:46:28.208072 4682 heartbeater.cc:464] Registering TS with master...
I20260927 12:46:28.208637 4682 heartbeater.cc:511] Master 127.0.98.62:36697 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 12:46:28.210481 4492 ts_manager.cc:194] Registered new tserver with Master: 946836d1a14e43ce82c08e8b6bcf41cd (127.0.98.1:35695)
I20260927 12:46:28.211755 4492 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60457
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 12:46:28.226959 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36697
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:46:28.337553 4687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:46:28.337814 4687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:46:28.337859 4687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:46:28.341827 4687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:46:28.341894 4687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:46:28.341965 4687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:46:28.347288 4687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:46:28.348338 4687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:46:28.349298 4687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:46:28.358700 4695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:28.358702 4692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:28.358731 4693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:28.359113 4687 server_base.cc:1034] running on GCE node
I20260927 12:46:28.359562 4687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:46:28.360221 4687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:46:28.361385 4687 hybrid_clock.cc:648] HybridClock initialized: now 1790513188361363 us; error 32 us; skew 500 ppm
Sep 27 12:46:28 dist-test-slave-lkn4 krb5kdc[4448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513188, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:46:28.364151 4687 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 12:46:28.364894 4687 fs_manager.cc:359] Metadata directory not provided
I20260927 12:46:28.364964 4687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:46:28.365154 4687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:46:28.367270 4687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/minicluster-data/ts-1/data/instance:
uuid: "80f9b90fb04641a8b7b3f569f3c7db2e"
format_stamp: "Formatted at 2026-09-27 12:46:28 on dist-test-slave-lkn4"
server_key: "2a53aefc030b4e68781d199dfab4c01b"
server_key_iv: "e430fb293f7ac1f935aea7f8c52af645"
server_key_version: "encryptionkey@0"
I20260927 12:46:28.367806 4687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/minicluster-data/ts-1/wal/instance:
uuid: "80f9b90fb04641a8b7b3f569f3c7db2e"
format_stamp: "Formatted at 2026-09-27 12:46:28 on dist-test-slave-lkn4"
server_key: "2a53aefc030b4e68781d199dfab4c01b"
server_key_iv: "e430fb293f7ac1f935aea7f8c52af645"
server_key_version: "encryptionkey@0"
I20260927 12:46:28.371443 4687 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 12:46:28.374013 4701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:28.375389 4687 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:46:28.375510 4687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/minicluster-data/ts-1/wal
uuid: "80f9b90fb04641a8b7b3f569f3c7db2e"
format_stamp: "Formatted at 2026-09-27 12:46:28 on dist-test-slave-lkn4"
server_key: "2a53aefc030b4e68781d199dfab4c01b"
server_key_iv: "e430fb293f7ac1f935aea7f8c52af645"
server_key_version: "encryptionkey@0"
I20260927 12:46:28.375614 4687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:46:28.387866 4687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:46:28.391418 4687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:46:28.391611 4687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:46:28.392200 4687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:46:28.393067 4687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:46:28.393114 4687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:28.393165 4687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:46:28.393194 4687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:28.401527 4687 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37205
I20260927 12:46:28.401588 4814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37205 every 8 connection(s)
I20260927 12:46:28.402170 4687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:46:28.403061 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 4687
I20260927 12:46:28.403164 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:46:28.403455 392 external_mini_cluster.cc:1428] Setting key 007984d629216442523733b7d09eea31
Sep 27 12:46:28 dist-test-slave-lkn4 krb5kdc[4448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513188, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 12:46:28.417208 4531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:28.403958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48005 (local address 127.0.98.62:36697)
0927 12:46:28.404097 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:28.404100 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:28.405040 (+ 940us) server_negotiation.cc:407] Connection header received
0927 12:46:28.406906 (+ 1866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:28.406910 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:28.406963 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:28.407050 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:28.408251 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:28.409045 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:28.410217 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:28.410555 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:28.410659 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:28.413472 (+ 2813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:28.413475 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:28.413477 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:28.413503 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:28.414983 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:28.415478 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:28.415482 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:28.415483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:28.415572 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:28.415967 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:28.415970 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:28.415972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:28.416206 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:28.416285 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:28.416928 (+ 643us) server_negotiation.cc:299] Negotiation successful
0927 12:46:28.417041 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 12:46:28.418154 4817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:28.404379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36697 (local address 127.0.98.2:48005)
0927 12:46:28.404912 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:28.404961 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:28.406573 (+ 1612us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:28.407228 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:28.407236 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:28.407608 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:28.408087 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:28.408101 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:28.409178 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:28.409181 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:28.410103 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:28.410113 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:28.410712 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:28.410715 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:28.411008 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:28.411612 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:28.411629 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:28.413298 (+ 1669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:28.415077 (+ 1779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:28.415082 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:28.415096 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:28.415349 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:28.415658 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:28.415661 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:28.415663 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:28.415865 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:28.416308 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:28.416315 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:28.416656 (+ 341us) client_negotiation.cc:769] Sending connection context
0927 12:46:28.416871 (+ 215us) client_negotiation.cc:240] Negotiation successful
0927 12:46:28.417249 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":144,"threads_started":1}
I20260927 12:46:28.419538 4815 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36697
I20260927 12:46:28.419804 4815 heartbeater.cc:464] Registering TS with master...
I20260927 12:46:28.420296 4815 heartbeater.cc:511] Master 127.0.98.62:36697 requested a full tablet report, sending...
I20260927 12:46:28.421432 4492 ts_manager.cc:194] Registered new tserver with Master: 80f9b90fb04641a8b7b3f569f3c7db2e (127.0.98.2:37205)
I20260927 12:46:28.422118 4492 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48005
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 12:46:28.430335 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36697
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:46:28.538002 4820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:46:28.538291 4820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:46:28.538353 4820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:46:28.542412 4820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:46:28.542478 4820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:46:28.542562 4820 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:46:28.547907 4820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:46:28.548969 4820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:46:28.549986 4820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:46:28.559301 4825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:28.559278 4826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:28.559334 4820 server_base.cc:1034] running on GCE node
W20260927 12:46:28.559664 4828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:28.560082 4820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:46:28.560817 4820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:46:28.562001 4820 hybrid_clock.cc:648] HybridClock initialized: now 1790513188561970 us; error 45 us; skew 500 ppm
Sep 27 12:46:28 dist-test-slave-lkn4 krb5kdc[4448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513188, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:46:28.565049 4820 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 12:46:28.565914 4820 fs_manager.cc:359] Metadata directory not provided
I20260927 12:46:28.565984 4820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:46:28.566205 4820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:46:28.568284 4820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/minicluster-data/ts-2/data/instance:
uuid: "e0582b5f536b45948283eec60fcefb6c"
format_stamp: "Formatted at 2026-09-27 12:46:28 on dist-test-slave-lkn4"
server_key: "5c046ce0ad8cfdcaab436e47fcfd2556"
server_key_iv: "ccd66321d1963d2739f16eda3f498000"
server_key_version: "encryptionkey@0"
I20260927 12:46:28.568859 4820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e0582b5f536b45948283eec60fcefb6c"
format_stamp: "Formatted at 2026-09-27 12:46:28 on dist-test-slave-lkn4"
server_key: "5c046ce0ad8cfdcaab436e47fcfd2556"
server_key_iv: "ccd66321d1963d2739f16eda3f498000"
server_key_version: "encryptionkey@0"
I20260927 12:46:28.572568 4820 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 12:46:28.575191 4834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:28.576469 4820 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:46:28.576577 4820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/minicluster-data/ts-2/wal
uuid: "e0582b5f536b45948283eec60fcefb6c"
format_stamp: "Formatted at 2026-09-27 12:46:28 on dist-test-slave-lkn4"
server_key: "5c046ce0ad8cfdcaab436e47fcfd2556"
server_key_iv: "ccd66321d1963d2739f16eda3f498000"
server_key_version: "encryptionkey@0"
I20260927 12:46:28.576674 4820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:46:28.594379 4820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:46:28.597646 4820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:46:28.597826 4820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:46:28.598486 4820 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:46:28.599325 4820 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:46:28.599371 4820 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:28.599414 4820 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:46:28.599443 4820 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:28.607407 4820 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37225
I20260927 12:46:28.607447 4947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37225 every 8 connection(s)
I20260927 12:46:28.607990 4820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:46:28.616293 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 4820
I20260927 12:46:28.616410 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:46:28.616676 392 external_mini_cluster.cc:1428] Setting key 762e46ca87a6d7e08169446dd6d70f7c
Sep 27 12:46:28 dist-test-slave-lkn4 krb5kdc[4448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513188, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:46:28.622292 4531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:28.609800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53133 (local address 127.0.98.62:36697)
0927 12:46:28.609963 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:28.609967 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:28.610893 (+ 926us) server_negotiation.cc:407] Connection header received
0927 12:46:28.612335 (+ 1442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:28.612338 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:28.612427 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:28.612525 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:28.613838 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:28.614421 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:28.615679 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:28.615913 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:28.616026 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:28.618832 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:28.618837 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:28.618840 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:28.618871 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:28.620268 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:28.620836 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:28.620839 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:28.620840 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:28.620932 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:28.621248 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:28.621251 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:28.621253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:28.621447 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:28.621527 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:28.622026 (+ 499us) server_negotiation.cc:299] Negotiation successful
0927 12:46:28.622166 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 12:46:28.623225 4950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:28.610223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36697 (local address 127.0.98.3:53133)
0927 12:46:28.610706 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:28.610784 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:28.612096 (+ 1312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:28.612700 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:28.612707 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:28.613146 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:28.613670 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:28.613684 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:28.614554 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:28.614557 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:28.615532 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:28.615542 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:28.616069 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:28.616072 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:28.616225 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:28.616850 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:28.616867 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:28.618650 (+ 1783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:28.620388 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:28.620393 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:28.620409 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:28.620686 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:28.621009 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:28.621011 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:28.621013 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:28.621151 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:28.621528 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:28.621533 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:28.621805 (+ 272us) client_negotiation.cc:769] Sending connection context
0927 12:46:28.621969 (+ 164us) client_negotiation.cc:240] Negotiation successful
0927 12:46:28.622300 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":125,"threads_started":1}
I20260927 12:46:28.624396 4948 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36697
I20260927 12:46:28.624634 4948 heartbeater.cc:464] Registering TS with master...
I20260927 12:46:28.625073 4948 heartbeater.cc:511] Master 127.0.98.62:36697 requested a full tablet report, sending...
I20260927 12:46:28.626088 4492 ts_manager.cc:194] Registered new tserver with Master: e0582b5f536b45948283eec60fcefb6c (127.0.98.3:37225)
I20260927 12:46:28.626673 4492 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53133
I20260927 12:46:28.630702 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 12:46:28.639935 4531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:28.632595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32926 (local address 127.0.98.62:36697)
0927 12:46:28.632701 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:28.632705 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:28.632870 (+ 165us) server_negotiation.cc:407] Connection header received
0927 12:46:28.632969 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:28.632971 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:28.633011 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:28.633075 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:28.633772 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:28.634354 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:28.635152 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:28.635367 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:28.635447 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:28.636868 (+ 1421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:28.636871 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:28.636874 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:28.636898 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:28.638313 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:28.638785 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:28.638787 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:28.638788 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:28.638861 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:28.639111 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:28.639116 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:28.639118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:28.639386 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:28.639505 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:28.639687 (+ 182us) server_negotiation.cc:299] Negotiation successful
0927 12:46:28.639801 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 12:46:28.643172 4492 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32926:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:46:28.646200 4492 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:46:28.661339 4960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:28.655204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51898 (local address 127.0.98.3:37225)
0927 12:46:28.656560 (+ 1356us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:28.656573 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:28.656596 (+ 23us) server_negotiation.cc:407] Connection header received
0927 12:46:28.656674 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:28.656680 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:28.656879 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:28.657061 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:46:28.657219 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:28.658190 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:28.659996 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:28.660661 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:28.660758 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:28.660928 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:28.661059 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 12:46:28.661158 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1259,"thread_start_us":104,"threads_started":1}
I20260927 12:46:28.661473 4961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:28.655327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37205 (local address 127.0.0.1:54336)
0927 12:46:28.656263 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:28.656276 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:28.656359 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:28.656884 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:28.656890 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:28.656911 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:46:28.657045 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:28.657051 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:28.658469 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:28.658472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:28.659851 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:28.659881 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:28.660844 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:28.660848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:28.660973 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:28.660997 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 12:46:28.661053 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 12:46:28.661316 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":876,"mutex_wait_us":198,"thread_start_us":88,"threads_started":1}
I20260927 12:46:28.661883 4962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:28.655039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37225 (local address 127.0.0.1:51898)
0927 12:46:28.656062 (+ 1023us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:28.656126 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:28.656260 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:28.657047 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:28.657050 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:28.657065 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:46:28.657136 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:28.657142 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:28.658344 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:28.658346 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:28.659860 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:28.659866 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:28.660767 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:28.660771 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:28.660886 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:28.660924 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 12:46:28.661030 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 12:46:28.661307 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":842,"mutex_wait_us":105,"thread_start_us":105,"threads_started":1}
I20260927 12:46:28.661971 4963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:28.655785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33644 (local address 127.0.98.1:35695)
0927 12:46:28.656123 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:28.656132 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:28.656611 (+ 479us) server_negotiation.cc:407] Connection header received
0927 12:46:28.656686 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:28.656691 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:28.656879 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:28.657038 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:46:28.657474 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:28.658583 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:28.659858 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:28.660710 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:28.660968 (+ 258us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:28.661125 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:28.661541 (+ 416us) server_negotiation.cc:299] Negotiation successful
0927 12:46:28.661642 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":55,"threads_started":1}
I20260927 12:46:28.661983 4964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:28.655893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54336 (local address 127.0.98.2:37205)
0927 12:46:28.656400 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:28.656412 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:28.656435 (+ 23us) server_negotiation.cc:407] Connection header received
0927 12:46:28.656510 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:28.656516 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:28.656745 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:28.656882 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:46:28.657176 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:28.658351 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:28.659957 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:28.660717 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:28.661173 (+ 456us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:28.661709 (+ 536us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:28.661757 (+ 48us) server_negotiation.cc:299] Negotiation successful
0927 12:46:28.661848 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":125,"threads_started":1}
I20260927 12:46:28.662817 4959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:28.655602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35695 (local address 127.0.0.1:33644)
0927 12:46:28.656510 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:28.656521 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:28.656591 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:28.657211 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:28.657215 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:28.657233 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:46:28.657345 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:28.657353 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:28.658735 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:28.658738 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:28.659741 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:28.659751 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:28.661181 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:28.661185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:28.661287 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:28.661306 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 12:46:28.661373 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 12:46:28.661433 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":858,"mutex_wait_us":3,"thread_start_us":108,"threads_started":1}
I20260927 12:46:28.665010 4616 tablet_service.cc:1467] Processing CreateTablet for tablet fc47598c7afb4b7ba85d8fbb7e27dde9 (DEFAULT_TABLE table=test-table [id=4e7fdaf2a87747e0ba469462b2383690]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:46:28.665027 4882 tablet_service.cc:1467] Processing CreateTablet for tablet fc47598c7afb4b7ba85d8fbb7e27dde9 (DEFAULT_TABLE table=test-table [id=4e7fdaf2a87747e0ba469462b2383690]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:46:28.665027 4749 tablet_service.cc:1467] Processing CreateTablet for tablet fc47598c7afb4b7ba85d8fbb7e27dde9 (DEFAULT_TABLE table=test-table [id=4e7fdaf2a87747e0ba469462b2383690]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:46:28.666071 4616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc47598c7afb4b7ba85d8fbb7e27dde9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:46:28.666085 4882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc47598c7afb4b7ba85d8fbb7e27dde9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:46:28.666080 4749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc47598c7afb4b7ba85d8fbb7e27dde9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:46:28.672008 4965 tablet_bootstrap.cc:492] T fc47598c7afb4b7ba85d8fbb7e27dde9 P 946836d1a14e43ce82c08e8b6bcf41cd: Bootstrap starting.
I20260927 12:46:28.673624 4967 tablet_bootstrap.cc:492] T fc47598c7afb4b7ba85d8fbb7e27dde9 P 80f9b90fb04641a8b7b3f569f3c7db2e: Bootstrap starting.
I20260927 12:46:28.674376 4966 tablet_bootstrap.cc:492] T fc47598c7afb4b7ba85d8fbb7e27dde9 P e0582b5f536b45948283eec60fcefb6c: Bootstrap starting.
I20260927 12:46:28.674548 4965 tablet_bootstrap.cc:654] T fc47598c7afb4b7ba85d8fbb7e27dde9 P 946836d1a14e43ce82c08e8b6bcf41cd: Neither blocks nor log segments found. Creating new log.
I20260927 12:46:28.675851 4965 log.cc:824] T fc47598c7afb4b7ba85d8fbb7e27dde9 P 946836d1a14e43ce82c08e8b6bcf41cd: Log is configured to *not* fsync() on all Append() calls
I20260927 12:46:28.675913 4967 tablet_bootstrap.cc:654] T fc47598c7afb4b7ba85d8fbb7e27dde9 P 80f9b90fb04641a8b7b3f569f3c7db2e: Neither blocks nor log segments found. Creating new log.
I20260927 12:46:28.676167 4966 tablet_bootstrap.cc:654] T fc47598c7afb4b7ba85d8fbb7e27dde9 P e0582b5f536b45948283eec60fcefb6c: Neither blocks nor log segments found. Creating new log.
I20260927 12:46:28.677078 4967 log.cc:824] T fc47598c7afb4b7ba85d8fbb7e27dde9 P 80f9b90fb04641a8b7b3f569f3c7db2e: Log is configured to *not* fsync() on all Append() calls
I20260927 12:46:28.677270 4966 log.cc:824] T fc47598c7afb4b7ba85d8fbb7e27dde9 P e0582b5f536b45948283eec60fcefb6c: Log is configured to *not* fsync() on all Append() calls
I20260927 12:46:28.678256 4965 tablet_bootstrap.cc:492] T fc47598c7afb4b7ba85d8fbb7e27dde9 P 946836d1a14e43ce82c08e8b6bcf41cd: No bootstrap required, opened a new log
I20260927 12:46:28.678443 4965 ts_tablet_manager.cc:1397] T fc47598c7afb4b7ba85d8fbb7e27dde9 P 946836d1a14e43ce82c08e8b6bcf41cd: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260927 12:46:28.679052 4967 tablet_bootstrap.cc:492] T fc47598c7afb4b7ba85d8fbb7e27dde9 P 80f9b90fb04641a8b7b3f569f3c7db2e: No bootstrap required, opened a new log
I20260927 12:46:28.679052 4966 tablet_bootstrap.cc:492] T fc47598c7afb4b7ba85d8fbb7e27dde9 P e0582b5f536b45948283eec60fcefb6c: No bootstrap required, opened a new log
I20260927 12:46:28.679239 4966 ts_tablet_manager.cc:1397] T fc47598c7afb4b7ba85d8fbb7e27dde9 P e0582b5f536b45948283eec60fcefb6c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 12:46:28.679239 4967 ts_tablet_manager.cc:1397] T fc47598c7afb4b7ba85d8fbb7e27dde9 P 80f9b90fb04641a8b7b3f569f3c7db2e: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 12:46:28.681890 4965 raft_consensus.cc:348] T fc47598c7afb4b7ba85d8fbb7e27dde9 P 946836d1a14e43ce82c08e8b6bcf41cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0582b5f536b45948283eec60fcefb6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37225 } } peers { permanent_uuid: "80f9b90fb04641a8b7b3f569f3c7db2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37205 } } peers { permanent_uuid: "946836d1a14e43ce82c08e8b6bcf41cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35695 } }
I20260927 12:46:28.682211 4965 raft_consensus.cc:374] T fc47598c7afb4b7ba85d8fbb7e27dde9 P 946836d1a14e43ce82c08e8b6bcf41cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:46:28.682264 4965 raft_consensus.cc:729] T fc47598c7afb4b7ba85d8fbb7e27dde9 P 946836d1a14e43ce82c08e8b6bcf41cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 946836d1a14e43ce82c08e8b6bcf41cd, State: Initialized, Role: FOLLOWER
I20260927 12:46:28.682693 4965 consensus_queue.cc:260] T fc47598c7afb4b7ba85d8fbb7e27dde9 P 946836d1a14e43ce82c08e8b6bcf41cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0582b5f536b45948283eec60fcefb6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37225 } } peers { permanent_uuid: "80f9b90fb04641a8b7b3f569f3c7db2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37205 } } peers { permanent_uuid: "946836d1a14e43ce82c08e8b6bcf41cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35695 } }
I20260927 12:46:28.682971 4967 raft_consensus.cc:348] T fc47598c7afb4b7ba85d8fbb7e27dde9 P 80f9b90fb04641a8b7b3f569f3c7db2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0582b5f536b45948283eec60fcefb6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37225 } } peers { permanent_uuid: "80f9b90fb04641a8b7b3f569f3c7db2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37205 } } peers { permanent_uuid: "946836d1a14e43ce82c08e8b6bcf41cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35695 } }
I20260927 12:46:28.683244 4967 raft_consensus.cc:374] T fc47598c7afb4b7ba85d8fbb7e27dde9 P 80f9b90fb04641a8b7b3f569f3c7db2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:46:28.683291 4967 raft_consensus.cc:729] T fc47598c7afb4b7ba85d8fbb7e27dde9 P 80f9b90fb04641a8b7b3f569f3c7db2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80f9b90fb04641a8b7b3f569f3c7db2e, State: Initialized, Role: FOLLOWER
I20260927 12:46:28.683310 4682 heartbeater.cc:503] Master 127.0.98.62:36697 was elected leader, sending a full tablet report...
I20260927 12:46:28.683352 4966 raft_consensus.cc:348] T fc47598c7afb4b7ba85d8fbb7e27dde9 P e0582b5f536b45948283eec60fcefb6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0582b5f536b45948283eec60fcefb6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37225 } } peers { permanent_uuid: "80f9b90fb04641a8b7b3f569f3c7db2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37205 } } peers { permanent_uuid: "946836d1a14e43ce82c08e8b6bcf41cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35695 } }
I20260927 12:46:28.683584 4966 raft_consensus.cc:374] T fc47598c7afb4b7ba85d8fbb7e27dde9 P e0582b5f536b45948283eec60fcefb6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:46:28.683627 4966 raft_consensus.cc:729] T fc47598c7afb4b7ba85d8fbb7e27dde9 P e0582b5f536b45948283eec60fcefb6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0582b5f536b45948283eec60fcefb6c, State: Initialized, Role: FOLLOWER
I20260927 12:46:28.683693 4965 ts_tablet_manager.cc:1428] T fc47598c7afb4b7ba85d8fbb7e27dde9 P 946836d1a14e43ce82c08e8b6bcf41cd: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 12:46:28.683705 4967 consensus_queue.cc:260] T fc47598c7afb4b7ba85d8fbb7e27dde9 P 80f9b90fb04641a8b7b3f569f3c7db2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0582b5f536b45948283eec60fcefb6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37225 } } peers { permanent_uuid: "80f9b90fb04641a8b7b3f569f3c7db2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37205 } } peers { permanent_uuid: "946836d1a14e43ce82c08e8b6bcf41cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35695 } }
I20260927 12:46:28.684005 4966 consensus_queue.cc:260] T fc47598c7afb4b7ba85d8fbb7e27dde9 P e0582b5f536b45948283eec60fcefb6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0582b5f536b45948283eec60fcefb6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37225 } } peers { permanent_uuid: "80f9b90fb04641a8b7b3f569f3c7db2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37205 } } peers { permanent_uuid: "946836d1a14e43ce82c08e8b6bcf41cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35695 } }
I20260927 12:46:28.684640 4967 ts_tablet_manager.cc:1428] T fc47598c7afb4b7ba85d8fbb7e27dde9 P 80f9b90fb04641a8b7b3f569f3c7db2e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 12:46:28.684973 4966 ts_tablet_manager.cc:1428] T fc47598c7afb4b7ba85d8fbb7e27dde9 P e0582b5f536b45948283eec60fcefb6c: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20260927 12:46:28.685000 4815 heartbeater.cc:503] Master 127.0.98.62:36697 was elected leader, sending a full tablet report...
I20260927 12:46:28.686338 4948 heartbeater.cc:503] Master 127.0.98.62:36697 was elected leader, sending a full tablet report...
W20260927 12:46:28.693215 4683 tablet.cc:2369] T fc47598c7afb4b7ba85d8fbb7e27dde9 P 946836d1a14e43ce82c08e8b6bcf41cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:46:28.827226 4971 raft_consensus.cc:482] T fc47598c7afb4b7ba85d8fbb7e27dde9 P 946836d1a14e43ce82c08e8b6bcf41cd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:46:28.827440 4971 raft_consensus.cc:504] T fc47598c7afb4b7ba85d8fbb7e27dde9 P 946836d1a14e43ce82c08e8b6bcf41cd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0582b5f536b45948283eec60fcefb6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37225 } } peers { permanent_uuid: "80f9b90fb04641a8b7b3f569f3c7db2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37205 } } peers { permanent_uuid: "946836d1a14e43ce82c08e8b6bcf41cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35695 } }
I20260927 12:46:28.828593 4971 leader_election.cc:288] T fc47598c7afb4b7ba85d8fbb7e27dde9 P 946836d1a14e43ce82c08e8b6bcf41cd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0582b5f536b45948283eec60fcefb6c (127.0.98.3:37225), 80f9b90fb04641a8b7b3f569f3c7db2e (127.0.98.2:37205)
I20260927 12:46:28.832551 4960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:28.828993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42543 (local address 127.0.98.3:37225)
0927 12:46:28.829121 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:28.829125 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:28.829436 (+ 311us) server_negotiation.cc:407] Connection header received
0927 12:46:28.829548 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:28.829550 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:28.829592 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:28.829660 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:46:28.829943 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:28.830473 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:28.831576 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:28.831999 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:28.832077 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:28.832139 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:28.832334 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 12:46:28.832429 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 12:46:28.832589 4975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:28.828901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37205 (local address 127.0.98.1:33477)
0927 12:46:28.829248 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:28.829260 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:28.829379 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:28.829659 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:28.829662 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:28.829677 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:46:28.829833 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:28.829839 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:28.830544 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:28.830547 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:28.831519 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:28.831530 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:28.832138 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:28.832142 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:28.832246 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:28.832262 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 12:46:28.832337 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 12:46:28.832415 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":23,"spinlock_wait_cycles":1920,"thread_start_us":111,"threads_started":1}
I20260927 12:46:28.832580 4964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:28.829004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33477 (local address 127.0.98.2:37205)
0927 12:46:28.829120 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:28.829125 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:28.829354 (+ 229us) server_negotiation.cc:407] Connection header received
0927 12:46:28.829512 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:28.829515 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:28.829570 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:28.829625 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:46:28.829948 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:28.830453 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:28.831638 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:28.832030 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:28.832120 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:28.832177 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:28.832396 (+ 219us) server_negotiation.cc:299] Negotiation successful
0927 12:46:28.832466 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 12:46:28.832962 4974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:28.828903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37225 (local address 127.0.98.1:42543)
0927 12:46:28.829348 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:28.829360 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:28.829457 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:28.829681 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:28.829684 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:28.829721 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:46:28.829834 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:28.829839 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:28.830571 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:28.830573 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:28.831476 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:28.831485 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:28.832113 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:28.832117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:28.832215 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:28.832235 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 12:46:28.832297 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 12:46:28.832378 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"mutex_wait_us":48,"spinlock_wait_cycles":24832,"thread_start_us":113,"threads_started":1}
I20260927 12:46:28.833339 4769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc47598c7afb4b7ba85d8fbb7e27dde9" candidate_uuid: "946836d1a14e43ce82c08e8b6bcf41cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80f9b90fb04641a8b7b3f569f3c7db2e" is_pre_election: true
I20260927 12:46:28.833525 4902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc47598c7afb4b7ba85d8fbb7e27dde9" candidate_uuid: "946836d1a14e43ce82c08e8b6bcf41cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0582b5f536b45948283eec60fcefb6c" is_pre_election: true
I20260927 12:46:28.833642 4769 raft_consensus.cc:2454] T fc47598c7afb4b7ba85d8fbb7e27dde9 P 80f9b90fb04641a8b7b3f569f3c7db2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 946836d1a14e43ce82c08e8b6bcf41cd in term 0.
I20260927 12:46:28.833859 4902 raft_consensus.cc:2454] T fc47598c7afb4b7ba85d8fbb7e27dde9 P e0582b5f536b45948283eec60fcefb6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 946836d1a14e43ce82c08e8b6bcf41cd in term 0.
I20260927 12:46:28.834197 4569 leader_election.cc:302] T fc47598c7afb4b7ba85d8fbb7e27dde9 P 946836d1a14e43ce82c08e8b6bcf41cd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80f9b90fb04641a8b7b3f569f3c7db2e, 946836d1a14e43ce82c08e8b6bcf41cd; no voters:
I20260927 12:46:28.834493 4971 raft_consensus.cc:2781] T fc47598c7afb4b7ba85d8fbb7e27dde9 P 946836d1a14e43ce82c08e8b6bcf41cd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:46:28.834568 4971 raft_consensus.cc:482] T fc47598c7afb4b7ba85d8fbb7e27dde9 P 946836d1a14e43ce82c08e8b6bcf41cd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:46:28.834597 4971 raft_consensus.cc:3037] T fc47598c7afb4b7ba85d8fbb7e27dde9 P 946836d1a14e43ce82c08e8b6bcf41cd [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:46:28.835686 4971 raft_consensus.cc:504] T fc47598c7afb4b7ba85d8fbb7e27dde9 P 946836d1a14e43ce82c08e8b6bcf41cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0582b5f536b45948283eec60fcefb6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37225 } } peers { permanent_uuid: "80f9b90fb04641a8b7b3f569f3c7db2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37205 } } peers { permanent_uuid: "946836d1a14e43ce82c08e8b6bcf41cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35695 } }
I20260927 12:46:28.836017 4971 leader_election.cc:288] T fc47598c7afb4b7ba85d8fbb7e27dde9 P 946836d1a14e43ce82c08e8b6bcf41cd [CANDIDATE]: Term 1 election: Requested vote from peers e0582b5f536b45948283eec60fcefb6c (127.0.98.3:37225), 80f9b90fb04641a8b7b3f569f3c7db2e (127.0.98.2:37205)
I20260927 12:46:28.836328 4902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc47598c7afb4b7ba85d8fbb7e27dde9" candidate_uuid: "946836d1a14e43ce82c08e8b6bcf41cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0582b5f536b45948283eec60fcefb6c"
I20260927 12:46:28.836382 4769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc47598c7afb4b7ba85d8fbb7e27dde9" candidate_uuid: "946836d1a14e43ce82c08e8b6bcf41cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80f9b90fb04641a8b7b3f569f3c7db2e"
I20260927 12:46:28.836450 4902 raft_consensus.cc:3037] T fc47598c7afb4b7ba85d8fbb7e27dde9 P e0582b5f536b45948283eec60fcefb6c [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:46:28.836478 4769 raft_consensus.cc:3037] T fc47598c7afb4b7ba85d8fbb7e27dde9 P 80f9b90fb04641a8b7b3f569f3c7db2e [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:46:28.837687 4769 raft_consensus.cc:2454] T fc47598c7afb4b7ba85d8fbb7e27dde9 P 80f9b90fb04641a8b7b3f569f3c7db2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 946836d1a14e43ce82c08e8b6bcf41cd in term 1.
I20260927 12:46:28.837759 4902 raft_consensus.cc:2454] T fc47598c7afb4b7ba85d8fbb7e27dde9 P e0582b5f536b45948283eec60fcefb6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 946836d1a14e43ce82c08e8b6bcf41cd in term 1.
I20260927 12:46:28.837992 4570 leader_election.cc:302] T fc47598c7afb4b7ba85d8fbb7e27dde9 P 946836d1a14e43ce82c08e8b6bcf41cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 946836d1a14e43ce82c08e8b6bcf41cd, e0582b5f536b45948283eec60fcefb6c; no voters:
I20260927 12:46:28.838186 4971 raft_consensus.cc:2781] T fc47598c7afb4b7ba85d8fbb7e27dde9 P 946836d1a14e43ce82c08e8b6bcf41cd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:46:28.838400 4971 raft_consensus.cc:686] T fc47598c7afb4b7ba85d8fbb7e27dde9 P 946836d1a14e43ce82c08e8b6bcf41cd [term 1 LEADER]: Becoming Leader. State: Replica: 946836d1a14e43ce82c08e8b6bcf41cd, State: Running, Role: LEADER
I20260927 12:46:28.838742 4971 consensus_queue.cc:237] T fc47598c7afb4b7ba85d8fbb7e27dde9 P 946836d1a14e43ce82c08e8b6bcf41cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0582b5f536b45948283eec60fcefb6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37225 } } peers { permanent_uuid: "80f9b90fb04641a8b7b3f569f3c7db2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37205 } } peers { permanent_uuid: "946836d1a14e43ce82c08e8b6bcf41cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35695 } }
I20260927 12:46:28.842056 4492 catalog_manager.cc:5495] T fc47598c7afb4b7ba85d8fbb7e27dde9 P 946836d1a14e43ce82c08e8b6bcf41cd reported cstate change: term changed from 0 to 1, leader changed from <none> to 946836d1a14e43ce82c08e8b6bcf41cd (127.0.98.1). New cstate: current_term: 1 leader_uuid: "946836d1a14e43ce82c08e8b6bcf41cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0582b5f536b45948283eec60fcefb6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80f9b90fb04641a8b7b3f569f3c7db2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "946836d1a14e43ce82c08e8b6bcf41cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35695 } health_report { overall_health: HEALTHY } } }
W20260927 12:46:28.858795 4949 tablet.cc:2369] T fc47598c7afb4b7ba85d8fbb7e27dde9 P e0582b5f536b45948283eec60fcefb6c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:46:28.861025 4963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:28.857437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33654 (local address 127.0.98.1:35695)
0927 12:46:28.857556 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:28.857560 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:28.857575 (+ 15us) server_negotiation.cc:407] Connection header received
0927 12:46:28.857726 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:28.857730 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:28.857798 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:28.857893 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:46:28.858154 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:28.858659 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:28.859520 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:28.859771 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:28.859857 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:28.860146 (+ 289us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:46:28.860637 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:46:28.860742 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:28.860836 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 12:46:28.860893 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 12:46:28.867048 4902 raft_consensus.cc:1261] T fc47598c7afb4b7ba85d8fbb7e27dde9 P e0582b5f536b45948283eec60fcefb6c [term 1 FOLLOWER]: Refusing update from remote peer 946836d1a14e43ce82c08e8b6bcf41cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:46:28.867048 4769 raft_consensus.cc:1261] T fc47598c7afb4b7ba85d8fbb7e27dde9 P 80f9b90fb04641a8b7b3f569f3c7db2e [term 1 FOLLOWER]: Refusing update from remote peer 946836d1a14e43ce82c08e8b6bcf41cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:46:28.867836 4976 consensus_queue.cc:1035] T fc47598c7afb4b7ba85d8fbb7e27dde9 P 946836d1a14e43ce82c08e8b6bcf41cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0582b5f536b45948283eec60fcefb6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:46:28.868011 4971 consensus_queue.cc:1035] T fc47598c7afb4b7ba85d8fbb7e27dde9 P 946836d1a14e43ce82c08e8b6bcf41cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "80f9b90fb04641a8b7b3f569f3c7db2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:46:28.875901 4979 mvcc.cc:205] Tried to move back new op lower bound from 7333942021591425024 to 7333942021486264320. Current Snapshot: MvccSnapshot[applied={T|T < 7333942021591425024}]
I20260927 12:46:28.878721 4981 mvcc.cc:205] Tried to move back new op lower bound from 7333942021591425024 to 7333942021486264320. Current Snapshot: MvccSnapshot[applied={T|T < 7333942021591425024}]
I20260927 12:46:28.884012 4491 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32926:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:46:28.884160 4491 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32926:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:46:28.886396 4491 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f2d0840784584dd1a6361c075eda2075: Sending DeleteTablet for 3 replicas of tablet fc47598c7afb4b7ba85d8fbb7e27dde9
I20260927 12:46:28.887118 4749 tablet_service.cc:1514] Processing DeleteTablet for tablet fc47598c7afb4b7ba85d8fbb7e27dde9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:46:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54336
I20260927 12:46:28.887145 4882 tablet_service.cc:1514] Processing DeleteTablet for tablet fc47598c7afb4b7ba85d8fbb7e27dde9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:46:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51898
I20260927 12:46:28.887189 4616 tablet_service.cc:1514] Processing DeleteTablet for tablet fc47598c7afb4b7ba85d8fbb7e27dde9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:46:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33644
I20260927 12:46:28.888063 4989 tablet_replica.cc:330] T fc47598c7afb4b7ba85d8fbb7e27dde9 P e0582b5f536b45948283eec60fcefb6c: stopping tablet replica
I20260927 12:46:28.888063 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 4554
I20260927 12:46:28.888429 4989 raft_consensus.cc:2229] T fc47598c7afb4b7ba85d8fbb7e27dde9 P e0582b5f536b45948283eec60fcefb6c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:46:28.888729 4988 tablet_replica.cc:330] T fc47598c7afb4b7ba85d8fbb7e27dde9 P 80f9b90fb04641a8b7b3f569f3c7db2e: stopping tablet replica
I20260927 12:46:28.888923 4989 raft_consensus.cc:2258] T fc47598c7afb4b7ba85d8fbb7e27dde9 P e0582b5f536b45948283eec60fcefb6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:46:28.889047 4988 raft_consensus.cc:2229] T fc47598c7afb4b7ba85d8fbb7e27dde9 P 80f9b90fb04641a8b7b3f569f3c7db2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:46:28.890713 4989 ts_tablet_manager.cc:1905] T fc47598c7afb4b7ba85d8fbb7e27dde9 P e0582b5f536b45948283eec60fcefb6c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:46:28.893141 4989 ts_tablet_manager.cc:1918] T fc47598c7afb4b7ba85d8fbb7e27dde9 P e0582b5f536b45948283eec60fcefb6c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:46:28.893245 4989 log.cc:1196] T fc47598c7afb4b7ba85d8fbb7e27dde9 P e0582b5f536b45948283eec60fcefb6c: Deleting WAL directory at /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790513161160093-392-0/minicluster-data/ts-2/wal/wals/fc47598c7afb4b7ba85d8fbb7e27dde9
I20260927 12:46:28.893549 4989 ts_tablet_manager.cc:1939] T fc47598c7afb4b7ba85d8fbb7e27dde9 P e0582b5f536b45948283eec60fcefb6c: Deleting consensus metadata
I20260927 12:46:28.894857 4477 catalog_manager.cc:4841] TS e0582b5f536b45948283eec60fcefb6c (127.0.98.3:37225): tablet fc47598c7afb4b7ba85d8fbb7e27dde9 (table test-table [id=4e7fdaf2a87747e0ba469462b2383690]) successfully deleted
I20260927 12:46:28.894968 4988 raft_consensus.cc:2258] T fc47598c7afb4b7ba85d8fbb7e27dde9 P 80f9b90fb04641a8b7b3f569f3c7db2e [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 12:46:28.896338 4478 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35695 (error 108)
I20260927 12:46:28.896555 4988 ts_tablet_manager.cc:1905] T fc47598c7afb4b7ba85d8fbb7e27dde9 P 80f9b90fb04641a8b7b3f569f3c7db2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:46:28.897105 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 4687
I20260927 12:46:28.897485 4959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:28.897130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35695 (local address 127.0.0.1:33658)
0927 12:46:28.897339 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:28.897403 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
W20260927 12:46:28.897969 4478 catalog_manager.cc:4584] TS 946836d1a14e43ce82c08e8b6bcf41cd (127.0.98.1:35695): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fc47598c7afb4b7ba85d8fbb7e27dde9: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35695: connect: Connection refused (error 111)
I20260927 12:46:28.902212 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 4820
I20260927 12:46:28.902940 4959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:28.902395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37205 (local address 127.0.0.1:54344)
0927 12:46:28.902807 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:28.902860 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:37205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336}
I20260927 12:46:28.908648 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 4461
2026-09-27T12:46:28Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2281 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-27T12:46:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:46:28Z Disabled control of system clock
I20260927 12:46:28.936937 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790513161160093-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790513161160093-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790513161160093-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43921
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43921
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 12:46:29.049506 4995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:46:29.049785 4995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:46:29.049822 4995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:46:29.054090 4995 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:46:29.054162 4995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:46:29.054177 4995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:46:29.054193 4995 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:46:29.054207 4995 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:46:29.059222 4995 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790513161160093-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790513161160093-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43921
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790513161160093-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:46:29.060364 4995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:46:29.061807 4995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:46:29.070223 5000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:29.070319 5001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:29.070377 4995 server_base.cc:1034] running on GCE node
W20260927 12:46:29.070276 5003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:29.071071 4995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:46:29.072185 4995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:46:29.073364 4995 hybrid_clock.cc:648] HybridClock initialized: now 1790513189073346 us; error 36 us; skew 500 ppm
I20260927 12:46:29.075323 4995 webserver.cc:460] Webserver started at http://127.0.98.62:37201/ using document root <none> and password file <none>
I20260927 12:46:29.075917 4995 fs_manager.cc:359] Metadata directory not provided
I20260927 12:46:29.075980 4995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:46:29.076179 4995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:46:29.078245 4995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790513161160093-392-0/minicluster-data/master-0/data/instance:
uuid: "78d9de49e4d546baa7f00d915c567940"
format_stamp: "Formatted at 2026-09-27 12:46:29 on dist-test-slave-lkn4"
server_key: "f10716b30b49dd53f06ef165f38ac70f"
server_key_iv: "62d6973c70eebb055f499ef8752682c3"
server_key_version: "encryptionkey@0"
I20260927 12:46:29.078816 4995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790513161160093-392-0/minicluster-data/master-0/wal/instance:
uuid: "78d9de49e4d546baa7f00d915c567940"
format_stamp: "Formatted at 2026-09-27 12:46:29 on dist-test-slave-lkn4"
server_key: "f10716b30b49dd53f06ef165f38ac70f"
server_key_iv: "62d6973c70eebb055f499ef8752682c3"
server_key_version: "encryptionkey@0"
I20260927 12:46:29.082647 4995 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 12:46:29.085143 5009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:29.086529 4995 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:46:29.086648 4995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790513161160093-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790513161160093-392-0/minicluster-data/master-0/wal
uuid: "78d9de49e4d546baa7f00d915c567940"
format_stamp: "Formatted at 2026-09-27 12:46:29 on dist-test-slave-lkn4"
server_key: "f10716b30b49dd53f06ef165f38ac70f"
server_key_iv: "62d6973c70eebb055f499ef8752682c3"
server_key_version: "encryptionkey@0"
I20260927 12:46:29.086755 4995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790513161160093-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790513161160093-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790513161160093-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:46:29.091610 4995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:46:29.092429 4995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:46:29.092602 4995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:46:29.099471 4995 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43921
I20260927 12:46:29.099504 5061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43921 every 8 connection(s)
I20260927 12:46:29.100409 4995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:46:29.102398 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 4995
I20260927 12:46:29.102506 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790513161160093-392-0/minicluster-data/master-0/wal/instance
I20260927 12:46:29.102761 392 external_mini_cluster.cc:1428] Setting key db2d3c992163f779da44db4fd9a0ed25
I20260927 12:46:29.104588 5062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:46:29.108256 5065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:29.104031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48788 (local address 127.0.98.62:43921)
0927 12:46:29.104659 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:29.104678 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:29.104724 (+ 46us) server_negotiation.cc:407] Connection header received
0927 12:46:29.105457 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:29.105477 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:29.105865 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:29.106224 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:29.107009 (+ 785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:29.107013 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:29.107025 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:46:29.107035 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:29.107090 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:46:29.107106 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:29.107233 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:29.107302 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 12:46:29.107543 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":105,"threads_started":1}
I20260927 12:46:29.110484 5062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78d9de49e4d546baa7f00d915c567940: Bootstrap starting.
I20260927 12:46:29.112581 5062 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78d9de49e4d546baa7f00d915c567940: Neither blocks nor log segments found. Creating new log.
I20260927 12:46:29.113277 5062 log.cc:824] T 00000000000000000000000000000000 P 78d9de49e4d546baa7f00d915c567940: Log is configured to *not* fsync() on all Append() calls
I20260927 12:46:29.115229 5062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78d9de49e4d546baa7f00d915c567940: No bootstrap required, opened a new log
I20260927 12:46:29.118860 5062 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78d9de49e4d546baa7f00d915c567940 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d9de49e4d546baa7f00d915c567940" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43921 } }
I20260927 12:46:29.119094 5062 raft_consensus.cc:374] T 00000000000000000000000000000000 P 78d9de49e4d546baa7f00d915c567940 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:46:29.119132 5062 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78d9de49e4d546baa7f00d915c567940 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78d9de49e4d546baa7f00d915c567940, State: Initialized, Role: FOLLOWER
I20260927 12:46:29.119568 5062 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78d9de49e4d546baa7f00d915c567940 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d9de49e4d546baa7f00d915c567940" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43921 } }
I20260927 12:46:29.119710 5062 raft_consensus.cc:388] T 00000000000000000000000000000000 P 78d9de49e4d546baa7f00d915c567940 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:46:29.119771 5062 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78d9de49e4d546baa7f00d915c567940 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:46:29.119875 5062 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78d9de49e4d546baa7f00d915c567940 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:46:29.120764 5062 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78d9de49e4d546baa7f00d915c567940 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d9de49e4d546baa7f00d915c567940" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43921 } }
I20260927 12:46:29.121125 5062 leader_election.cc:302] T 00000000000000000000000000000000 P 78d9de49e4d546baa7f00d915c567940 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 78d9de49e4d546baa7f00d915c567940; no voters:
I20260927 12:46:29.121456 5062 leader_election.cc:288] T 00000000000000000000000000000000 P 78d9de49e4d546baa7f00d915c567940 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:46:29.121549 5067 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 78d9de49e4d546baa7f00d915c567940 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:46:29.121757 5067 raft_consensus.cc:686] T 00000000000000000000000000000000 P 78d9de49e4d546baa7f00d915c567940 [term 1 LEADER]: Becoming Leader. State: Replica: 78d9de49e4d546baa7f00d915c567940, State: Running, Role: LEADER
I20260927 12:46:29.122095 5067 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78d9de49e4d546baa7f00d915c567940 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d9de49e4d546baa7f00d915c567940" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43921 } }
I20260927 12:46:29.122691 5062 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78d9de49e4d546baa7f00d915c567940 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:46:29.123740 5068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78d9de49e4d546baa7f00d915c567940 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "78d9de49e4d546baa7f00d915c567940" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d9de49e4d546baa7f00d915c567940" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43921 } } }
I20260927 12:46:29.123780 5069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78d9de49e4d546baa7f00d915c567940 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78d9de49e4d546baa7f00d915c567940. Latest consensus state: current_term: 1 leader_uuid: "78d9de49e4d546baa7f00d915c567940" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78d9de49e4d546baa7f00d915c567940" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43921 } } }
I20260927 12:46:29.123883 5068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78d9de49e4d546baa7f00d915c567940 [sys.catalog]: This master's current role is: LEADER
I20260927 12:46:29.123883 5069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78d9de49e4d546baa7f00d915c567940 [sys.catalog]: This master's current role is: LEADER
I20260927 12:46:29.124267 5076 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:46:29.127339 5076 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:46:29.133014 5076 catalog_manager.cc:1347] Generated new cluster ID: 1a46b68816ec42609c61164e41199b2c
I20260927 12:46:29.133100 5076 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:46:29.143165 5076 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:46:29.143502 5076 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:46:29.150661 5076 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 78d9de49e4d546baa7f00d915c567940: Generated new TSK 0
I20260927 12:46:29.151369 5076 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:46:29.163676 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790513161160093-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790513161160093-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790513161160093-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43921
--builtin_ntp_servers=127.0.98.20:44597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 12:46:29.275547 5086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:46:29.275835 5086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:46:29.275875 5086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:46:29.280061 5086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:46:29.280135 5086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:46:29.280195 5086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:46:29.285149 5086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790513161160093-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790513161160093-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790513161160093-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:46:29.286387 5086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:46:29.287827 5086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 12:46:29.298168 5086 server_base.cc:1034] running on GCE node
W20260927 12:46:29.298126 5091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:29.298126 5092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:29.298123 5094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:29.298825 5086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:46:29.299496 5086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:46:29.300688 5086 hybrid_clock.cc:648] HybridClock initialized: now 1790513189300659 us; error 75 us; skew 500 ppm
I20260927 12:46:29.308251 5086 webserver.cc:460] Webserver started at http://127.0.98.1:44209/ using document root <none> and password file <none>
I20260927 12:46:29.308928 5086 fs_manager.cc:359] Metadata directory not provided
I20260927 12:46:29.309003 5086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:46:29.309196 5086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:46:29.311384 5086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790513161160093-392-0/minicluster-data/ts-0/data/instance:
uuid: "36940050e94a4be2b9c6fa9dde7ef999"
format_stamp: "Formatted at 2026-09-27 12:46:29 on dist-test-slave-lkn4"
server_key: "b95339721857681c2a6e5409f34fcec9"
server_key_iv: "cea60e24c6cc908ef8dc4d29159af197"
server_key_version: "encryptionkey@0"
I20260927 12:46:29.312114 5086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790513161160093-392-0/minicluster-data/ts-0/wal/instance:
uuid: "36940050e94a4be2b9c6fa9dde7ef999"
format_stamp: "Formatted at 2026-09-27 12:46:29 on dist-test-slave-lkn4"
server_key: "b95339721857681c2a6e5409f34fcec9"
server_key_iv: "cea60e24c6cc908ef8dc4d29159af197"
server_key_version: "encryptionkey@0"
I20260927 12:46:29.315945 5086 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 12:46:29.318455 5100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:29.319768 5086 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:46:29.319864 5086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790513161160093-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790513161160093-392-0/minicluster-data/ts-0/wal
uuid: "36940050e94a4be2b9c6fa9dde7ef999"
format_stamp: "Formatted at 2026-09-27 12:46:29 on dist-test-slave-lkn4"
server_key: "b95339721857681c2a6e5409f34fcec9"
server_key_iv: "cea60e24c6cc908ef8dc4d29159af197"
server_key_version: "encryptionkey@0"
I20260927 12:46:29.319955 5086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790513161160093-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790513161160093-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790513161160093-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:46:29.324649 5086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:46:29.325273 5086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:46:29.325433 5086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:46:29.325937 5086 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:46:29.326861 5086 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:46:29.326906 5086 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:29.326942 5086 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:46:29.326961 5086 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:29.335034 5086 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34979
I20260927 12:46:29.335058 5213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34979 every 8 connection(s)
I20260927 12:46:29.335935 5086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:46:29.339674 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 5086
I20260927 12:46:29.339808 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790513161160093-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:46:29.340109 392 external_mini_cluster.cc:1428] Setting key 93791358327d423600447e23d965e4e3
I20260927 12:46:29.342128 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790513161160093-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790513161160093-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790513161160093-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43921
--builtin_ntp_servers=127.0.98.20:44597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 12:46:29.342549 5065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:29.337773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53107 (local address 127.0.98.62:43921)
0927 12:46:29.337938 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:29.337941 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:29.339170 (+ 1229us) server_negotiation.cc:407] Connection header received
0927 12:46:29.340716 (+ 1546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:29.340720 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:29.340776 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:29.340853 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:29.342090 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:29.342094 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:29.342097 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:46:29.342111 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:29.342144 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:46:29.342153 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:29.342261 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:29.342397 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 12:46:29.342438 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 12:46:29.343719 5216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:29.338269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43921 (local address 127.0.98.1:53107)
0927 12:46:29.339006 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:29.339049 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:29.340517 (+ 1468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:29.341028 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:29.341035 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:29.341127 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:29.341731 (+ 604us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 12:46:29.341733 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:29.341796 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 12:46:29.341888 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:29.342242 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:29.342251 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:29.342257 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 12:46:29.342439 (+ 182us) client_negotiation.cc:240] Negotiation successful
0927 12:46:29.342702 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":114,"threads_started":1}
I20260927 12:46:29.344950 5214 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43921
I20260927 12:46:29.345244 5214 heartbeater.cc:464] Registering TS with master...
I20260927 12:46:29.345711 5214 heartbeater.cc:511] Master 127.0.98.62:43921 requested a full tablet report, sending...
I20260927 12:46:29.347334 5026 ts_manager.cc:194] Registered new tserver with Master: 36940050e94a4be2b9c6fa9dde7ef999 (127.0.98.1:34979)
W20260927 12:46:29.454979 5217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:46:29.455251 5217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:46:29.455295 5217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:46:29.459332 5217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:46:29.459386 5217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:46:29.459451 5217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:46:29.464277 5217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790513161160093-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790513161160093-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790513161160093-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:46:29.465375 5217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:46:29.466960 5217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 12:46:29.476552 5217 server_base.cc:1034] running on GCE node
W20260927 12:46:29.476511 5222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:29.476539 5225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:29.476511 5223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:29.477387 5217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:46:29.478088 5217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:46:29.479271 5217 hybrid_clock.cc:648] HybridClock initialized: now 1790513189479249 us; error 34 us; skew 500 ppm
I20260927 12:46:29.481151 5217 webserver.cc:460] Webserver started at http://127.0.98.2:39753/ using document root <none> and password file <none>
I20260927 12:46:29.481736 5217 fs_manager.cc:359] Metadata directory not provided
I20260927 12:46:29.481796 5217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:46:29.481978 5217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:46:29.484035 5217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790513161160093-392-0/minicluster-data/ts-1/data/instance:
uuid: "a22526c80e9b4f51a94caf18d8321f4f"
format_stamp: "Formatted at 2026-09-27 12:46:29 on dist-test-slave-lkn4"
server_key: "4f3f0c103b383feafc887120c781786d"
server_key_iv: "4c4300da2cbc80903e1b1a8899049683"
server_key_version: "encryptionkey@0"
I20260927 12:46:29.484551 5217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790513161160093-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a22526c80e9b4f51a94caf18d8321f4f"
format_stamp: "Formatted at 2026-09-27 12:46:29 on dist-test-slave-lkn4"
server_key: "4f3f0c103b383feafc887120c781786d"
server_key_iv: "4c4300da2cbc80903e1b1a8899049683"
server_key_version: "encryptionkey@0"
I20260927 12:46:29.488268 5217 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260927 12:46:29.490816 5231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:29.492127 5217 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 12:46:29.492266 5217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790513161160093-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790513161160093-392-0/minicluster-data/ts-1/wal
uuid: "a22526c80e9b4f51a94caf18d8321f4f"
format_stamp: "Formatted at 2026-09-27 12:46:29 on dist-test-slave-lkn4"
server_key: "4f3f0c103b383feafc887120c781786d"
server_key_iv: "4c4300da2cbc80903e1b1a8899049683"
server_key_version: "encryptionkey@0"
I20260927 12:46:29.492396 5217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790513161160093-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790513161160093-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790513161160093-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:46:29.497098 5217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:46:29.497664 5217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:46:29.497808 5217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:46:29.498387 5217 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:46:29.499284 5217 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:46:29.499326 5217 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:29.499363 5217 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:46:29.499389 5217 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:29.507586 5217 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45007
I20260927 12:46:29.507607 5344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45007 every 8 connection(s)
I20260927 12:46:29.508515 5217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:46:29.508993 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 5217
I20260927 12:46:29.509114 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790513161160093-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:46:29.509395 392 external_mini_cluster.cc:1428] Setting key 6515263a111215c0d6a25b0aedab5247
I20260927 12:46:29.511714 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790513161160093-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790513161160093-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790513161160093-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43921
--builtin_ntp_servers=127.0.98.20:44597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 12:46:29.515527 5065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:29.510392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47901 (local address 127.0.98.62:43921)
0927 12:46:29.510530 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:29.510534 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:29.512045 (+ 1511us) server_negotiation.cc:407] Connection header received
0927 12:46:29.513698 (+ 1653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:29.513700 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:29.513738 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:29.513817 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:29.515019 (+ 1202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:29.515022 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:29.515025 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:46:29.515040 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:29.515063 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:46:29.515074 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:29.515173 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:29.515339 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 12:46:29.515386 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 12:46:29.516462 5347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:29.510957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43921 (local address 127.0.98.2:47901)
0927 12:46:29.511878 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:29.511936 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:29.513435 (+ 1499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:29.514003 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:29.514010 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:29.514149 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:29.514755 (+ 606us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 12:46:29.514756 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:29.514798 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 12:46:29.514867 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:29.515191 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:29.515200 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:29.515206 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 12:46:29.515357 (+ 151us) client_negotiation.cc:240] Negotiation successful
0927 12:46:29.515561 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":140,"threads_started":1}
I20260927 12:46:29.517532 5345 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43921
I20260927 12:46:29.517802 5345 heartbeater.cc:464] Registering TS with master...
I20260927 12:46:29.518311 5345 heartbeater.cc:511] Master 127.0.98.62:43921 requested a full tablet report, sending...
I20260927 12:46:29.519454 5026 ts_manager.cc:194] Registered new tserver with Master: a22526c80e9b4f51a94caf18d8321f4f (127.0.98.2:45007)
W20260927 12:46:29.621582 5348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:46:29.621879 5348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:46:29.621934 5348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:46:29.626345 5348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:46:29.626413 5348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:46:29.626494 5348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:46:29.631397 5348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790513161160093-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790513161160093-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5348
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790513161160093-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:46:29.632517 5348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:46:29.633908 5348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:46:29.643827 5354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:29.643891 5356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:29.643838 5353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:29.644095 5348 server_base.cc:1034] running on GCE node
I20260927 12:46:29.644589 5348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:46:29.645325 5348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:46:29.646504 5348 hybrid_clock.cc:648] HybridClock initialized: now 1790513189646480 us; error 40 us; skew 500 ppm
I20260927 12:46:29.648555 5348 webserver.cc:460] Webserver started at http://127.0.98.3:37899/ using document root <none> and password file <none>
I20260927 12:46:29.649156 5348 fs_manager.cc:359] Metadata directory not provided
I20260927 12:46:29.649232 5348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:46:29.649456 5348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:46:29.652132 5348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790513161160093-392-0/minicluster-data/ts-2/data/instance:
uuid: "ac5c2af110f446e3be3af9d73eceb5f3"
format_stamp: "Formatted at 2026-09-27 12:46:29 on dist-test-slave-lkn4"
server_key: "505c9ea1c19dab1a3e347c83292449dc"
server_key_iv: "289509e36367a408cea40cdee184d836"
server_key_version: "encryptionkey@0"
I20260927 12:46:29.652812 5348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790513161160093-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ac5c2af110f446e3be3af9d73eceb5f3"
format_stamp: "Formatted at 2026-09-27 12:46:29 on dist-test-slave-lkn4"
server_key: "505c9ea1c19dab1a3e347c83292449dc"
server_key_iv: "289509e36367a408cea40cdee184d836"
server_key_version: "encryptionkey@0"
I20260927 12:46:29.657060 5348 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 12:46:29.659876 5362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:29.661129 5348 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 12:46:29.661275 5348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790513161160093-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790513161160093-392-0/minicluster-data/ts-2/wal
uuid: "ac5c2af110f446e3be3af9d73eceb5f3"
format_stamp: "Formatted at 2026-09-27 12:46:29 on dist-test-slave-lkn4"
server_key: "505c9ea1c19dab1a3e347c83292449dc"
server_key_iv: "289509e36367a408cea40cdee184d836"
server_key_version: "encryptionkey@0"
I20260927 12:46:29.661384 5348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790513161160093-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790513161160093-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790513161160093-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:46:29.665836 5348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:46:29.666474 5348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:46:29.666631 5348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:46:29.667145 5348 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:46:29.668006 5348 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:46:29.668057 5348 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:29.668105 5348 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:46:29.668130 5348 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:29.676270 5348 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43309
I20260927 12:46:29.676302 5475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43309 every 8 connection(s)
I20260927 12:46:29.677197 5348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:46:29.677907 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 5348
I20260927 12:46:29.678011 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790513161160093-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:46:29.678347 392 external_mini_cluster.cc:1428] Setting key 7a76b48bebb78130141e56a9030e63f6
I20260927 12:46:29.683382 5065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:29.679072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40003 (local address 127.0.98.62:43921)
0927 12:46:29.679195 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:29.679200 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:29.680493 (+ 1293us) server_negotiation.cc:407] Connection header received
0927 12:46:29.681756 (+ 1263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:29.681759 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:29.681796 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:29.681883 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:29.683011 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:29.683014 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:29.683016 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:46:29.683026 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:29.683048 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:46:29.683056 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:29.683127 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:29.683243 (+ 116us) server_negotiation.cc:299] Negotiation successful
0927 12:46:29.683278 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 12:46:29.684321 5478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:29.679620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43921 (local address 127.0.98.3:40003)
0927 12:46:29.680338 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:29.680393 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:29.681541 (+ 1148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:29.682092 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:29.682099 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:29.682192 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:29.682741 (+ 549us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 12:46:29.682743 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:29.682787 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 12:46:29.682873 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:29.683122 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:29.683126 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:29.683130 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 12:46:29.683251 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 12:46:29.683459 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":108,"threads_started":1}
I20260927 12:46:29.685356 5476 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43921
I20260927 12:46:29.685573 5476 heartbeater.cc:464] Registering TS with master...
I20260927 12:46:29.685969 5476 heartbeater.cc:511] Master 127.0.98.62:43921 requested a full tablet report, sending...
I20260927 12:46:29.687007 5026 ts_manager.cc:194] Registered new tserver with Master: ac5c2af110f446e3be3af9d73eceb5f3 (127.0.98.3:43309)
I20260927 12:46:29.691982 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 12:46:29.695508 5065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:29.693876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48796 (local address 127.0.98.62:43921)
0927 12:46:29.694022 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:29.694026 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:29.694131 (+ 105us) server_negotiation.cc:407] Connection header received
0927 12:46:29.694279 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:29.694283 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:29.694320 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:29.694383 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:29.695138 (+ 755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:29.695141 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:29.695142 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:46:29.695153 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:29.695173 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:46:29.695181 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:29.695260 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:29.695326 (+ 66us) server_negotiation.cc:299] Negotiation successful
0927 12:46:29.695369 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 12:46:29.698623 5026 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48796:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:46:29.701647 5026 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:46:29.713352 5487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:29.710265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34979 (local address 127.0.0.1:33664)
0927 12:46:29.710945 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:29.710998 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:29.711120 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:29.711629 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:29.711635 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:29.711715 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:29.712315 (+ 600us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 12:46:29.712317 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:29.712338 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 12:46:29.712422 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:29.713016 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:29.713024 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:29.713031 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 12:46:29.713137 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 12:46:29.713204 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"spinlock_wait_cycles":7680,"thread_start_us":118,"threads_started":1}
I20260927 12:46:29.713379 5489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:29.710404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33664 (local address 127.0.98.1:34979)
0927 12:46:29.711175 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:29.711191 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:29.711209 (+ 18us) server_negotiation.cc:407] Connection header received
0927 12:46:29.711270 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:29.711274 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:29.711467 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:29.712001 (+ 534us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:29.712852 (+ 851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:29.712856 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:29.712865 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:46:29.712873 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:29.712902 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:46:29.712918 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:29.713044 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:29.713159 (+ 115us) server_negotiation.cc:299] Negotiation successful
0927 12:46:29.713230 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":676,"thread_start_us":117,"threads_started":1}
I20260927 12:46:29.713896 5491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:29.711092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43309 (local address 127.0.0.1:49358)
0927 12:46:29.711606 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:29.711621 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:29.711703 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:29.712444 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:29.712447 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:29.712463 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:29.712897 (+ 434us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 12:46:29.712899 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:29.712912 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 12:46:29.712942 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:29.713677 (+ 735us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:29.713681 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:29.713685 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 12:46:29.713744 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 12:46:29.713780 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":92,"threads_started":1}
I20260927 12:46:29.714427 5492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:29.711405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49358 (local address 127.0.98.3:43309)
0927 12:46:29.711955 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:29.711970 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:29.711993 (+ 23us) server_negotiation.cc:407] Connection header received
0927 12:46:29.712066 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:29.712071 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:29.712277 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:29.712701 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:29.713468 (+ 767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:29.713474 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:29.713485 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:46:29.713497 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:29.713544 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:46:29.713569 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:29.714131 (+ 562us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:29.714183 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 12:46:29.714260 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":100,"threads_started":1}
I20260927 12:46:29.715067 5488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:29.710335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45007 (local address 127.0.0.1:57946)
0927 12:46:29.710980 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:29.710997 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:29.711107 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:29.712572 (+ 1465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:29.712575 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:29.712590 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:29.713579 (+ 989us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 12:46:29.713581 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:29.713590 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 12:46:29.713615 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:29.713939 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:29.713942 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:29.714216 (+ 274us) client_negotiation.cc:769] Sending connection context
0927 12:46:29.714900 (+ 684us) client_negotiation.cc:240] Negotiation successful
0927 12:46:29.714938 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"spinlock_wait_cycles":3840,"thread_start_us":95,"threads_started":1}
I20260927 12:46:29.715493 5490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:29.710624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57946 (local address 127.0.98.2:45007)
0927 12:46:29.710977 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:29.710994 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:29.711207 (+ 213us) server_negotiation.cc:407] Connection header received
0927 12:46:29.711262 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:29.711268 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:29.711467 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:29.711880 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:29.713746 (+ 1866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:29.713751 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:29.713763 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:46:29.713775 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:29.713815 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:46:29.713849 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:29.713946 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:29.715245 (+ 1299us) server_negotiation.cc:299] Negotiation successful
0927 12:46:29.715325 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":78,"threads_started":1}
I20260927 12:46:29.716702 5148 tablet_service.cc:1467] Processing CreateTablet for tablet 125b737a46e047068519136f0f2da36d (DEFAULT_TABLE table=test-table [id=2d1b015d7a56431e9ebdd5ccefe3fcc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:46:29.716946 5410 tablet_service.cc:1467] Processing CreateTablet for tablet 125b737a46e047068519136f0f2da36d (DEFAULT_TABLE table=test-table [id=2d1b015d7a56431e9ebdd5ccefe3fcc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:46:29.717820 5410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 125b737a46e047068519136f0f2da36d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:46:29.717854 5148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 125b737a46e047068519136f0f2da36d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:46:29.718181 5279 tablet_service.cc:1467] Processing CreateTablet for tablet 125b737a46e047068519136f0f2da36d (DEFAULT_TABLE table=test-table [id=2d1b015d7a56431e9ebdd5ccefe3fcc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:46:29.719218 5279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 125b737a46e047068519136f0f2da36d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:46:29.724152 5493 tablet_bootstrap.cc:492] T 125b737a46e047068519136f0f2da36d P ac5c2af110f446e3be3af9d73eceb5f3: Bootstrap starting.
I20260927 12:46:29.726100 5494 tablet_bootstrap.cc:492] T 125b737a46e047068519136f0f2da36d P 36940050e94a4be2b9c6fa9dde7ef999: Bootstrap starting.
I20260927 12:46:29.726502 5493 tablet_bootstrap.cc:654] T 125b737a46e047068519136f0f2da36d P ac5c2af110f446e3be3af9d73eceb5f3: Neither blocks nor log segments found. Creating new log.
I20260927 12:46:29.726873 5495 tablet_bootstrap.cc:492] T 125b737a46e047068519136f0f2da36d P a22526c80e9b4f51a94caf18d8321f4f: Bootstrap starting.
I20260927 12:46:29.727711 5493 log.cc:824] T 125b737a46e047068519136f0f2da36d P ac5c2af110f446e3be3af9d73eceb5f3: Log is configured to *not* fsync() on all Append() calls
I20260927 12:46:29.728345 5494 tablet_bootstrap.cc:654] T 125b737a46e047068519136f0f2da36d P 36940050e94a4be2b9c6fa9dde7ef999: Neither blocks nor log segments found. Creating new log.
I20260927 12:46:29.728986 5495 tablet_bootstrap.cc:654] T 125b737a46e047068519136f0f2da36d P a22526c80e9b4f51a94caf18d8321f4f: Neither blocks nor log segments found. Creating new log.
I20260927 12:46:29.729198 5494 log.cc:824] T 125b737a46e047068519136f0f2da36d P 36940050e94a4be2b9c6fa9dde7ef999: Log is configured to *not* fsync() on all Append() calls
I20260927 12:46:29.729460 5493 tablet_bootstrap.cc:492] T 125b737a46e047068519136f0f2da36d P ac5c2af110f446e3be3af9d73eceb5f3: No bootstrap required, opened a new log
I20260927 12:46:29.729641 5493 ts_tablet_manager.cc:1397] T 125b737a46e047068519136f0f2da36d P ac5c2af110f446e3be3af9d73eceb5f3: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 12:46:29.729795 5495 log.cc:824] T 125b737a46e047068519136f0f2da36d P a22526c80e9b4f51a94caf18d8321f4f: Log is configured to *not* fsync() on all Append() calls
I20260927 12:46:29.731205 5494 tablet_bootstrap.cc:492] T 125b737a46e047068519136f0f2da36d P 36940050e94a4be2b9c6fa9dde7ef999: No bootstrap required, opened a new log
I20260927 12:46:29.731400 5494 ts_tablet_manager.cc:1397] T 125b737a46e047068519136f0f2da36d P 36940050e94a4be2b9c6fa9dde7ef999: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 12:46:29.731719 5495 tablet_bootstrap.cc:492] T 125b737a46e047068519136f0f2da36d P a22526c80e9b4f51a94caf18d8321f4f: No bootstrap required, opened a new log
I20260927 12:46:29.731897 5495 ts_tablet_manager.cc:1397] T 125b737a46e047068519136f0f2da36d P a22526c80e9b4f51a94caf18d8321f4f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 12:46:29.733922 5493 raft_consensus.cc:348] T 125b737a46e047068519136f0f2da36d P ac5c2af110f446e3be3af9d73eceb5f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36940050e94a4be2b9c6fa9dde7ef999" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34979 } } peers { permanent_uuid: "a22526c80e9b4f51a94caf18d8321f4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 } } peers { permanent_uuid: "ac5c2af110f446e3be3af9d73eceb5f3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } }
I20260927 12:46:29.734205 5493 raft_consensus.cc:374] T 125b737a46e047068519136f0f2da36d P ac5c2af110f446e3be3af9d73eceb5f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:46:29.734258 5493 raft_consensus.cc:729] T 125b737a46e047068519136f0f2da36d P ac5c2af110f446e3be3af9d73eceb5f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac5c2af110f446e3be3af9d73eceb5f3, State: Initialized, Role: FOLLOWER
I20260927 12:46:29.734687 5493 consensus_queue.cc:260] T 125b737a46e047068519136f0f2da36d P ac5c2af110f446e3be3af9d73eceb5f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36940050e94a4be2b9c6fa9dde7ef999" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34979 } } peers { permanent_uuid: "a22526c80e9b4f51a94caf18d8321f4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 } } peers { permanent_uuid: "ac5c2af110f446e3be3af9d73eceb5f3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } }
I20260927 12:46:29.735306 5476 heartbeater.cc:503] Master 127.0.98.62:43921 was elected leader, sending a full tablet report...
I20260927 12:46:29.735733 5493 ts_tablet_manager.cc:1428] T 125b737a46e047068519136f0f2da36d P ac5c2af110f446e3be3af9d73eceb5f3: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 12:46:29.736059 5494 raft_consensus.cc:348] T 125b737a46e047068519136f0f2da36d P 36940050e94a4be2b9c6fa9dde7ef999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36940050e94a4be2b9c6fa9dde7ef999" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34979 } } peers { permanent_uuid: "a22526c80e9b4f51a94caf18d8321f4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 } } peers { permanent_uuid: "ac5c2af110f446e3be3af9d73eceb5f3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } }
I20260927 12:46:29.736346 5494 raft_consensus.cc:374] T 125b737a46e047068519136f0f2da36d P 36940050e94a4be2b9c6fa9dde7ef999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:46:29.736209 5495 raft_consensus.cc:348] T 125b737a46e047068519136f0f2da36d P a22526c80e9b4f51a94caf18d8321f4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36940050e94a4be2b9c6fa9dde7ef999" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34979 } } peers { permanent_uuid: "a22526c80e9b4f51a94caf18d8321f4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 } } peers { permanent_uuid: "ac5c2af110f446e3be3af9d73eceb5f3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } }
I20260927 12:46:29.736402 5494 raft_consensus.cc:729] T 125b737a46e047068519136f0f2da36d P 36940050e94a4be2b9c6fa9dde7ef999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36940050e94a4be2b9c6fa9dde7ef999, State: Initialized, Role: FOLLOWER
I20260927 12:46:29.736418 5495 raft_consensus.cc:374] T 125b737a46e047068519136f0f2da36d P a22526c80e9b4f51a94caf18d8321f4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:46:29.736469 5495 raft_consensus.cc:729] T 125b737a46e047068519136f0f2da36d P a22526c80e9b4f51a94caf18d8321f4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a22526c80e9b4f51a94caf18d8321f4f, State: Initialized, Role: FOLLOWER
I20260927 12:46:29.736877 5494 consensus_queue.cc:260] T 125b737a46e047068519136f0f2da36d P 36940050e94a4be2b9c6fa9dde7ef999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36940050e94a4be2b9c6fa9dde7ef999" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34979 } } peers { permanent_uuid: "a22526c80e9b4f51a94caf18d8321f4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 } } peers { permanent_uuid: "ac5c2af110f446e3be3af9d73eceb5f3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } }
I20260927 12:46:29.736877 5495 consensus_queue.cc:260] T 125b737a46e047068519136f0f2da36d P a22526c80e9b4f51a94caf18d8321f4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36940050e94a4be2b9c6fa9dde7ef999" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34979 } } peers { permanent_uuid: "a22526c80e9b4f51a94caf18d8321f4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 } } peers { permanent_uuid: "ac5c2af110f446e3be3af9d73eceb5f3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } }
I20260927 12:46:29.737744 5345 heartbeater.cc:503] Master 127.0.98.62:43921 was elected leader, sending a full tablet report...
I20260927 12:46:29.737877 5495 ts_tablet_manager.cc:1428] T 125b737a46e047068519136f0f2da36d P a22526c80e9b4f51a94caf18d8321f4f: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 12:46:29.737877 5494 ts_tablet_manager.cc:1428] T 125b737a46e047068519136f0f2da36d P 36940050e94a4be2b9c6fa9dde7ef999: Time spent starting tablet: real 0.006s user 0.004s sys 0.004s
I20260927 12:46:29.738283 5214 heartbeater.cc:503] Master 127.0.98.62:43921 was elected leader, sending a full tablet report...
W20260927 12:46:29.759588 5346 tablet.cc:2369] T 125b737a46e047068519136f0f2da36d P a22526c80e9b4f51a94caf18d8321f4f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 12:46:29.837095 5215 tablet.cc:2369] T 125b737a46e047068519136f0f2da36d P 36940050e94a4be2b9c6fa9dde7ef999: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:46:29.857947 5501 raft_consensus.cc:482] T 125b737a46e047068519136f0f2da36d P a22526c80e9b4f51a94caf18d8321f4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:46:29.858191 5501 raft_consensus.cc:504] T 125b737a46e047068519136f0f2da36d P a22526c80e9b4f51a94caf18d8321f4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36940050e94a4be2b9c6fa9dde7ef999" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34979 } } peers { permanent_uuid: "a22526c80e9b4f51a94caf18d8321f4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 } } peers { permanent_uuid: "ac5c2af110f446e3be3af9d73eceb5f3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } }
I20260927 12:46:29.859382 5501 leader_election.cc:288] T 125b737a46e047068519136f0f2da36d P a22526c80e9b4f51a94caf18d8321f4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36940050e94a4be2b9c6fa9dde7ef999 (127.0.98.1:34979), ac5c2af110f446e3be3af9d73eceb5f3 (127.0.98.3:43309)
I20260927 12:46:29.861680 5502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:29.859987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43309 (local address 127.0.98.2:51445)
0927 12:46:29.860371 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:29.860384 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:29.860476 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:29.860710 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:29.860711 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:29.860724 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:29.861069 (+ 345us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 12:46:29.861071 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:29.861078 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 12:46:29.861096 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:29.861366 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:29.861369 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:29.861372 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 12:46:29.861446 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 12:46:29.861509 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":127,"threads_started":1}
I20260927 12:46:29.861615 5489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:29.859898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58131 (local address 127.0.98.1:34979)
0927 12:46:29.860015 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:29.860020 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:29.860035 (+ 15us) server_negotiation.cc:407] Connection header received
0927 12:46:29.860083 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:29.860086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:29.860118 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:29.860217 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:29.861223 (+ 1006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:29.861227 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:29.861230 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:46:29.861243 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:29.861267 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:46:29.861277 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:29.861374 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:29.861444 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 12:46:29.861486 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 12:46:29.861765 5492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:29.860124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51445 (local address 127.0.98.3:43309)
0927 12:46:29.860289 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:29.860293 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:29.860457 (+ 164us) server_negotiation.cc:407] Connection header received
0927 12:46:29.860568 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:29.860571 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:29.860606 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:29.860687 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:29.861223 (+ 536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:29.861227 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:29.861230 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:46:29.861243 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:29.861266 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:46:29.861279 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:29.861383 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:29.861543 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 12:46:29.861581 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 12:46:29.862208 5347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:29.859712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34979 (local address 127.0.98.2:58131)
0927 12:46:29.859848 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:29.859864 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:29.859952 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:29.860313 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:29.860316 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:29.860331 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:29.861011 (+ 680us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 12:46:29.861013 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:29.861027 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 12:46:29.861059 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:29.861366 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:29.861369 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:29.861372 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 12:46:29.861446 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 12:46:29.862105 (+ 659us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":610}
I20260927 12:46:29.862540 5430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125b737a46e047068519136f0f2da36d" candidate_uuid: "a22526c80e9b4f51a94caf18d8321f4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac5c2af110f446e3be3af9d73eceb5f3" is_pre_election: true
I20260927 12:46:29.862810 5168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125b737a46e047068519136f0f2da36d" candidate_uuid: "a22526c80e9b4f51a94caf18d8321f4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36940050e94a4be2b9c6fa9dde7ef999" is_pre_election: true
I20260927 12:46:29.862850 5430 raft_consensus.cc:2454] T 125b737a46e047068519136f0f2da36d P ac5c2af110f446e3be3af9d73eceb5f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a22526c80e9b4f51a94caf18d8321f4f in term 0.
I20260927 12:46:29.863076 5168 raft_consensus.cc:2454] T 125b737a46e047068519136f0f2da36d P 36940050e94a4be2b9c6fa9dde7ef999 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a22526c80e9b4f51a94caf18d8321f4f in term 0.
I20260927 12:46:29.863293 5233 leader_election.cc:302] T 125b737a46e047068519136f0f2da36d P a22526c80e9b4f51a94caf18d8321f4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a22526c80e9b4f51a94caf18d8321f4f, ac5c2af110f446e3be3af9d73eceb5f3; no voters:
I20260927 12:46:29.863565 5501 raft_consensus.cc:2781] T 125b737a46e047068519136f0f2da36d P a22526c80e9b4f51a94caf18d8321f4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:46:29.863642 5501 raft_consensus.cc:482] T 125b737a46e047068519136f0f2da36d P a22526c80e9b4f51a94caf18d8321f4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:46:29.863673 5501 raft_consensus.cc:3037] T 125b737a46e047068519136f0f2da36d P a22526c80e9b4f51a94caf18d8321f4f [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:46:29.864804 5501 raft_consensus.cc:504] T 125b737a46e047068519136f0f2da36d P a22526c80e9b4f51a94caf18d8321f4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36940050e94a4be2b9c6fa9dde7ef999" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34979 } } peers { permanent_uuid: "a22526c80e9b4f51a94caf18d8321f4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 } } peers { permanent_uuid: "ac5c2af110f446e3be3af9d73eceb5f3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } }
I20260927 12:46:29.865159 5501 leader_election.cc:288] T 125b737a46e047068519136f0f2da36d P a22526c80e9b4f51a94caf18d8321f4f [CANDIDATE]: Term 1 election: Requested vote from peers 36940050e94a4be2b9c6fa9dde7ef999 (127.0.98.1:34979), ac5c2af110f446e3be3af9d73eceb5f3 (127.0.98.3:43309)
I20260927 12:46:29.865536 5168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125b737a46e047068519136f0f2da36d" candidate_uuid: "a22526c80e9b4f51a94caf18d8321f4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36940050e94a4be2b9c6fa9dde7ef999"
I20260927 12:46:29.865608 5430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "125b737a46e047068519136f0f2da36d" candidate_uuid: "a22526c80e9b4f51a94caf18d8321f4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac5c2af110f446e3be3af9d73eceb5f3"
I20260927 12:46:29.865650 5168 raft_consensus.cc:3037] T 125b737a46e047068519136f0f2da36d P 36940050e94a4be2b9c6fa9dde7ef999 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:46:29.865751 5430 raft_consensus.cc:3037] T 125b737a46e047068519136f0f2da36d P ac5c2af110f446e3be3af9d73eceb5f3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:46:29.866750 5168 raft_consensus.cc:2454] T 125b737a46e047068519136f0f2da36d P 36940050e94a4be2b9c6fa9dde7ef999 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a22526c80e9b4f51a94caf18d8321f4f in term 1.
I20260927 12:46:29.866891 5430 raft_consensus.cc:2454] T 125b737a46e047068519136f0f2da36d P ac5c2af110f446e3be3af9d73eceb5f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a22526c80e9b4f51a94caf18d8321f4f in term 1.
I20260927 12:46:29.867019 5233 leader_election.cc:302] T 125b737a46e047068519136f0f2da36d P a22526c80e9b4f51a94caf18d8321f4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36940050e94a4be2b9c6fa9dde7ef999, a22526c80e9b4f51a94caf18d8321f4f; no voters:
I20260927 12:46:29.867173 5501 raft_consensus.cc:2781] T 125b737a46e047068519136f0f2da36d P a22526c80e9b4f51a94caf18d8321f4f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:46:29.867390 5501 raft_consensus.cc:686] T 125b737a46e047068519136f0f2da36d P a22526c80e9b4f51a94caf18d8321f4f [term 1 LEADER]: Becoming Leader. State: Replica: a22526c80e9b4f51a94caf18d8321f4f, State: Running, Role: LEADER
I20260927 12:46:29.867779 5501 consensus_queue.cc:237] T 125b737a46e047068519136f0f2da36d P a22526c80e9b4f51a94caf18d8321f4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36940050e94a4be2b9c6fa9dde7ef999" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34979 } } peers { permanent_uuid: "a22526c80e9b4f51a94caf18d8321f4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 } } peers { permanent_uuid: "ac5c2af110f446e3be3af9d73eceb5f3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } }
I20260927 12:46:29.871120 5026 catalog_manager.cc:5495] T 125b737a46e047068519136f0f2da36d P a22526c80e9b4f51a94caf18d8321f4f reported cstate change: term changed from 0 to 1, leader changed from <none> to a22526c80e9b4f51a94caf18d8321f4f (127.0.98.2). New cstate: current_term: 1 leader_uuid: "a22526c80e9b4f51a94caf18d8321f4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36940050e94a4be2b9c6fa9dde7ef999" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a22526c80e9b4f51a94caf18d8321f4f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac5c2af110f446e3be3af9d73eceb5f3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 } health_report { overall_health: UNKNOWN } } }
I20260927 12:46:29.905392 5490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:29.904069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57960 (local address 127.0.98.2:45007)
0927 12:46:29.904193 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:29.904196 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:29.904208 (+ 12us) server_negotiation.cc:407] Connection header received
0927 12:46:29.904283 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:29.904286 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:29.904344 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:29.904413 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:29.905098 (+ 685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:29.905101 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:29.905103 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:46:29.905115 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:29.905134 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:46:29.905143 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:29.905225 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:29.905253 (+ 28us) server_negotiation.cc:299] Negotiation successful
0927 12:46:29.905287 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 12:46:29.911572 5168 raft_consensus.cc:1261] T 125b737a46e047068519136f0f2da36d P 36940050e94a4be2b9c6fa9dde7ef999 [term 1 FOLLOWER]: Refusing update from remote peer a22526c80e9b4f51a94caf18d8321f4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:46:29.911572 5430 raft_consensus.cc:1261] T 125b737a46e047068519136f0f2da36d P ac5c2af110f446e3be3af9d73eceb5f3 [term 1 FOLLOWER]: Refusing update from remote peer a22526c80e9b4f51a94caf18d8321f4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:46:29.912364 5503 consensus_queue.cc:1035] T 125b737a46e047068519136f0f2da36d P a22526c80e9b4f51a94caf18d8321f4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac5c2af110f446e3be3af9d73eceb5f3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:46:29.912576 5501 consensus_queue.cc:1035] T 125b737a46e047068519136f0f2da36d P a22526c80e9b4f51a94caf18d8321f4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "36940050e94a4be2b9c6fa9dde7ef999" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:46:29.920334 5506 mvcc.cc:205] Tried to move back new op lower bound from 7333942025870094336 to 7333942025701044224. Current Snapshot: MvccSnapshot[applied={T|T < 7333942025870094336}]
I20260927 12:46:29.920897 5508 mvcc.cc:205] Tried to move back new op lower bound from 7333942025870094336 to 7333942025701044224. Current Snapshot: MvccSnapshot[applied={T|T < 7333942025870094336}]
I20260927 12:46:29.921473 5507 mvcc.cc:205] Tried to move back new op lower bound from 7333942025870094336 to 7333942025701044224. Current Snapshot: MvccSnapshot[applied={T|T < 7333942025870094336}]
I20260927 12:46:29.931970 5025 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:46:29.932175 5025 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:46:29.934859 5025 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 78d9de49e4d546baa7f00d915c567940: Sending DeleteTablet for 3 replicas of tablet 125b737a46e047068519136f0f2da36d
I20260927 12:46:29.935824 5410 tablet_service.cc:1514] Processing DeleteTablet for tablet 125b737a46e047068519136f0f2da36d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:46:29 UTC) from {username='slave'} at 127.0.0.1:49358
I20260927 12:46:29.935791 5279 tablet_service.cc:1514] Processing DeleteTablet for tablet 125b737a46e047068519136f0f2da36d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:46:29 UTC) from {username='slave'} at 127.0.0.1:57946
I20260927 12:46:29.935818 5148 tablet_service.cc:1514] Processing DeleteTablet for tablet 125b737a46e047068519136f0f2da36d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:46:29 UTC) from {username='slave'} at 127.0.0.1:33664
I20260927 12:46:29.936409 5517 tablet_replica.cc:330] T 125b737a46e047068519136f0f2da36d P ac5c2af110f446e3be3af9d73eceb5f3: stopping tablet replica
I20260927 12:46:29.936600 5516 tablet_replica.cc:330] T 125b737a46e047068519136f0f2da36d P a22526c80e9b4f51a94caf18d8321f4f: stopping tablet replica
I20260927 12:46:29.936913 5517 raft_consensus.cc:2229] T 125b737a46e047068519136f0f2da36d P ac5c2af110f446e3be3af9d73eceb5f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:46:29.937089 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 5086
I20260927 12:46:29.937290 5517 raft_consensus.cc:2258] T 125b737a46e047068519136f0f2da36d P ac5c2af110f446e3be3af9d73eceb5f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:46:29.937107 5516 raft_consensus.cc:2229] T 125b737a46e047068519136f0f2da36d P a22526c80e9b4f51a94caf18d8321f4f [term 1 LEADER]: Raft consensus shutting down.
I20260927 12:46:29.939002 5516 raft_consensus.cc:2258] T 125b737a46e047068519136f0f2da36d P a22526c80e9b4f51a94caf18d8321f4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:46:29.939054 5517 ts_tablet_manager.cc:1905] T 125b737a46e047068519136f0f2da36d P ac5c2af110f446e3be3af9d73eceb5f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:46:29.940527 5516 ts_tablet_manager.cc:1905] T 125b737a46e047068519136f0f2da36d P a22526c80e9b4f51a94caf18d8321f4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:46:29.941766 5517 ts_tablet_manager.cc:1918] T 125b737a46e047068519136f0f2da36d P ac5c2af110f446e3be3af9d73eceb5f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:46:29.941860 5517 log.cc:1196] T 125b737a46e047068519136f0f2da36d P ac5c2af110f446e3be3af9d73eceb5f3: Deleting WAL directory at /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790513161160093-392-0/minicluster-data/ts-2/wal/wals/125b737a46e047068519136f0f2da36d
I20260927 12:46:29.942236 5517 ts_tablet_manager.cc:1939] T 125b737a46e047068519136f0f2da36d P ac5c2af110f446e3be3af9d73eceb5f3: Deleting consensus metadata
I20260927 12:46:29.943368 5516 ts_tablet_manager.cc:1918] T 125b737a46e047068519136f0f2da36d P a22526c80e9b4f51a94caf18d8321f4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:46:29.943521 5516 log.cc:1196] T 125b737a46e047068519136f0f2da36d P a22526c80e9b4f51a94caf18d8321f4f: Deleting WAL directory at /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790513161160093-392-0/minicluster-data/ts-1/wal/wals/125b737a46e047068519136f0f2da36d
I20260927 12:46:29.943570 5011 catalog_manager.cc:4841] TS ac5c2af110f446e3be3af9d73eceb5f3 (127.0.98.3:43309): tablet 125b737a46e047068519136f0f2da36d (table test-table [id=2d1b015d7a56431e9ebdd5ccefe3fcc9]) successfully deleted
I20260927 12:46:29.943910 5516 ts_tablet_manager.cc:1939] T 125b737a46e047068519136f0f2da36d P a22526c80e9b4f51a94caf18d8321f4f: Deleting consensus metadata
I20260927 12:46:29.945113 5013 catalog_manager.cc:4841] TS a22526c80e9b4f51a94caf18d8321f4f (127.0.98.2:45007): tablet 125b737a46e047068519136f0f2da36d (table test-table [id=2d1b015d7a56431e9ebdd5ccefe3fcc9]) successfully deleted
W20260927 12:46:29.945174 5011 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34979 (error 108)
I20260927 12:46:29.945839 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 5217
I20260927 12:46:29.946519 5488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:29.945915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34979 (local address 127.0.0.1:33668)
0927 12:46:29.946370 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:29.946438 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375}
W20260927 12:46:29.947337 5011 catalog_manager.cc:4584] TS 36940050e94a4be2b9c6fa9dde7ef999 (127.0.98.1:34979): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 125b737a46e047068519136f0f2da36d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34979: connect: Connection refused (error 111)
I20260927 12:46:29.950855 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 5348
I20260927 12:46:29.955370 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 4995
2026-09-27T12:46:29Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1045 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260927 12:46:29.975171 392 mini_oidc.cc:152] Starting JWKS server
I20260927 12:46:29.976775 392 webserver.cc:460] Webserver started at https://127.0.0.1:36989/ using document root <none> and password file <none>
I20260927 12:46:29.976939 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 12:46:29.977187 392 webserver.cc:460] Webserver started at http://127.0.0.1:40213/ using document root <none> and password file <none>
2026-09-27T12:46:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:46:29Z Disabled control of system clock
I20260927 12:46:29.985590 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790513161160093-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790513161160093-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790513161160093-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33765
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:41015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33765
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36989/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790513161160093-392-0/testchainca.cert with env {}
W20260927 12:46:30.094823 5525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:46:30.095118 5525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:46:30.095160 5525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:46:30.099303 5525 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:46:30.099370 5525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:46:30.099385 5525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:46:30.099400 5525 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:46:30.099413 5525 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 12:46:30.099431 5525 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 12:46:30.099455 5525 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36989/jwks.json
I20260927 12:46:30.104916 5525 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790513161160093-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790513161160093-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33765
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36989/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790513161160093-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5525
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790513161160093-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:46:30.106118 5525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:46:30.107187 5525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:46:30.116053 5531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:30.116070 5530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:30.116076 5533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:30.116129 5525 server_base.cc:1034] running on GCE node
I20260927 12:46:30.116912 5525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:46:30.118058 5525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:46:30.119256 5525 hybrid_clock.cc:648] HybridClock initialized: now 1790513190119237 us; error 30 us; skew 500 ppm
I20260927 12:46:30.121192 5525 webserver.cc:460] Webserver started at http://127.0.98.62:38327/ using document root <none> and password file <none>
I20260927 12:46:30.121753 5525 fs_manager.cc:359] Metadata directory not provided
I20260927 12:46:30.121819 5525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:46:30.122020 5525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:46:30.124177 5525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790513161160093-392-0/minicluster-data/master-0/data/instance:
uuid: "98a272be18f945b8b2c1976fb3cea422"
format_stamp: "Formatted at 2026-09-27 12:46:30 on dist-test-slave-lkn4"
server_key: "e5bec7e6ce9f3f57ac12c2b46530b2af"
server_key_iv: "f64f9350cc357c46777ac2a9e8119005"
server_key_version: "encryptionkey@0"
I20260927 12:46:30.124761 5525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790513161160093-392-0/minicluster-data/master-0/wal/instance:
uuid: "98a272be18f945b8b2c1976fb3cea422"
format_stamp: "Formatted at 2026-09-27 12:46:30 on dist-test-slave-lkn4"
server_key: "e5bec7e6ce9f3f57ac12c2b46530b2af"
server_key_iv: "f64f9350cc357c46777ac2a9e8119005"
server_key_version: "encryptionkey@0"
I20260927 12:46:30.128746 5525 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 12:46:30.131346 5539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:30.132704 5525 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 12:46:30.132838 5525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790513161160093-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790513161160093-392-0/minicluster-data/master-0/wal
uuid: "98a272be18f945b8b2c1976fb3cea422"
format_stamp: "Formatted at 2026-09-27 12:46:30 on dist-test-slave-lkn4"
server_key: "e5bec7e6ce9f3f57ac12c2b46530b2af"
server_key_iv: "f64f9350cc357c46777ac2a9e8119005"
server_key_version: "encryptionkey@0"
I20260927 12:46:30.132943 5525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790513161160093-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790513161160093-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790513161160093-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:46:30.155251 5525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:46:30.155992 5525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:46:30.156173 5525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:46:30.162999 5525 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33765
I20260927 12:46:30.163014 5591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33765 every 8 connection(s)
I20260927 12:46:30.163995 5525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:46:30.167358 5592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:46:30.171327 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 5525
I20260927 12:46:30.171430 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790513161160093-392-0/minicluster-data/master-0/wal/instance
I20260927 12:46:30.171665 392 external_mini_cluster.cc:1428] Setting key cf94edcce4b5157d8638e89e4f1a9885
I20260927 12:46:30.173020 5592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98a272be18f945b8b2c1976fb3cea422: Bootstrap starting.
I20260927 12:46:30.175364 5592 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98a272be18f945b8b2c1976fb3cea422: Neither blocks nor log segments found. Creating new log.
I20260927 12:46:30.176163 5592 log.cc:824] T 00000000000000000000000000000000 P 98a272be18f945b8b2c1976fb3cea422: Log is configured to *not* fsync() on all Append() calls
I20260927 12:46:30.178165 5592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98a272be18f945b8b2c1976fb3cea422: No bootstrap required, opened a new log
I20260927 12:46:30.180831 5595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:30.173080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49132 (local address 127.0.98.62:33765)
0927 12:46:30.173752 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:30.173770 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:30.173811 (+ 41us) server_negotiation.cc:407] Connection header received
0927 12:46:30.174502 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:30.174524 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:30.174988 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:30.175307 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:30.175731 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:30.176736 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:30.177643 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:30.178177 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:30.178330 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:30.179133 (+ 803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:30.179138 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:30.179151 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:46:30.179560 (+ 409us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:30.179613 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:46:30.179632 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:30.179714 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:30.179860 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 12:46:30.180109 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":135,"threads_started":1}
I20260927 12:46:30.181807 5592 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98a272be18f945b8b2c1976fb3cea422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98a272be18f945b8b2c1976fb3cea422" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33765 } }
I20260927 12:46:30.182096 5592 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98a272be18f945b8b2c1976fb3cea422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:46:30.182160 5592 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98a272be18f945b8b2c1976fb3cea422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98a272be18f945b8b2c1976fb3cea422, State: Initialized, Role: FOLLOWER
I20260927 12:46:30.182610 5592 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98a272be18f945b8b2c1976fb3cea422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98a272be18f945b8b2c1976fb3cea422" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33765 } }
I20260927 12:46:30.182765 5592 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98a272be18f945b8b2c1976fb3cea422 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:46:30.182816 5592 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98a272be18f945b8b2c1976fb3cea422 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:46:30.182909 5592 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98a272be18f945b8b2c1976fb3cea422 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:46:30.183707 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:46:30.183959 5592 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98a272be18f945b8b2c1976fb3cea422 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98a272be18f945b8b2c1976fb3cea422" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33765 } }
I20260927 12:46:30.184284 5592 leader_election.cc:302] T 00000000000000000000000000000000 P 98a272be18f945b8b2c1976fb3cea422 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98a272be18f945b8b2c1976fb3cea422; no voters:
I20260927 12:46:30.184571 5592 leader_election.cc:288] T 00000000000000000000000000000000 P 98a272be18f945b8b2c1976fb3cea422 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:46:30.184710 5597 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98a272be18f945b8b2c1976fb3cea422 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:46:30.184877 5597 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98a272be18f945b8b2c1976fb3cea422 [term 1 LEADER]: Becoming Leader. State: Replica: 98a272be18f945b8b2c1976fb3cea422, State: Running, Role: LEADER
I20260927 12:46:30.185276 5597 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98a272be18f945b8b2c1976fb3cea422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98a272be18f945b8b2c1976fb3cea422" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33765 } }
I20260927 12:46:30.186064 5592 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98a272be18f945b8b2c1976fb3cea422 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:46:30.186913 5599 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98a272be18f945b8b2c1976fb3cea422 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98a272be18f945b8b2c1976fb3cea422. Latest consensus state: current_term: 1 leader_uuid: "98a272be18f945b8b2c1976fb3cea422" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98a272be18f945b8b2c1976fb3cea422" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33765 } } }
I20260927 12:46:30.187026 5599 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98a272be18f945b8b2c1976fb3cea422 [sys.catalog]: This master's current role is: LEADER
I20260927 12:46:30.187407 5606 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:46:30.187515 5598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98a272be18f945b8b2c1976fb3cea422 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "98a272be18f945b8b2c1976fb3cea422" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98a272be18f945b8b2c1976fb3cea422" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33765 } } }
I20260927 12:46:30.187708 5598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98a272be18f945b8b2c1976fb3cea422 [sys.catalog]: This master's current role is: LEADER
I20260927 12:46:30.190572 5606 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:46:30.195452 5606 catalog_manager.cc:1347] Generated new cluster ID: c389903894bd46329c3b3133cccbf0b8
I20260927 12:46:30.195523 5606 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:46:30.227898 5606 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:46:30.228883 5606 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:46:30.235952 5606 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98a272be18f945b8b2c1976fb3cea422: Generated new TSK 0
I20260927 12:46:30.236673 5606 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 12:46:30.272867 5622 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260927 12:46:30.272969 5622 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 12:46:30.273584 5595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:30.256446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49140 (local address 127.0.98.62:33765)
0927 12:46:30.256568 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:30.256572 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:30.256724 (+ 152us) server_negotiation.cc:407] Connection header received
0927 12:46:30.256931 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:30.256935 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:30.256991 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:30.257099 (+ 108us) server_negotiation.cc:226] Negotiated authn=JWT
0927 12:46:30.257327 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:30.258153 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:30.259004 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:30.259359 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:30.259456 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:30.259797 (+ 341us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 12:46:30.273149 (+ 13352us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36989/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 12:46:30.273444 (+ 295us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49140: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36989/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":51}
W20260927 12:46:30.273670 5595 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49140: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36989/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260927 12:46:30.274884 5621 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:46:30.256346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33765 (local address 127.0.0.1:49140)
0927 12:46:30.256643 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:30.256656 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:30.256782 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:30.257101 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:30.257103 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:30.257113 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0927 12:46:30.257236 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:30.257241 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:30.258270 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:30.258273 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:30.258904 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:30.258912 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:30.259469 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:30.259487 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:30.259561 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:30.259605 (+ 44us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 12:46:30.273402 (+ 13797us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36989/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 12:46:30.273621 (+ 219us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:33765: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36989/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":90,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:33765: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36989/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260927 12:46:30.275632 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 5525
2026-09-27T12:46:30Z chronyd exiting
I20260927 12:46:30.312153 392 test_util.cc:175] -----------------------------------------------
I20260927 12:46:30.312250 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790513161160093-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (338 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260927 12:46:30.313315 392 mini_oidc.cc:152] Starting JWKS server
I20260927 12:46:30.314270 392 webserver.cc:460] Webserver started at https://127.0.0.1:36739/ using document root <none> and password file <none>
I20260927 12:46:30.314374 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 12:46:30.314792 392 webserver.cc:460] Webserver started at http://127.0.0.1:37753/ using document root <none> and password file <none>
2026-09-27T12:46:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:46:30Z Disabled control of system clock
I20260927 12:46:30.323233 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790513161160093-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790513161160093-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790513161160093-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40773
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40773
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36739/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790513161160093-392-0/testchainca.cert with env {}
W20260927 12:46:30.433346 5628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:46:30.433646 5628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:46:30.433687 5628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:46:30.437835 5628 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:46:30.437904 5628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:46:30.437918 5628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:46:30.437935 5628 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:46:30.437949 5628 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 12:46:30.437965 5628 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 12:46:30.437992 5628 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36739/jwks.json
I20260927 12:46:30.443809 5628 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790513161160093-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790513161160093-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40773
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36739/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790513161160093-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790513161160093-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:46:30.444934 5628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:46:30.445945 5628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 12:46:30.454542 5628 server_base.cc:1034] running on GCE node
W20260927 12:46:30.454519 5633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:30.454713 5634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:30.454520 5636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:30.455387 5628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:46:30.456545 5628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:46:30.457708 5628 hybrid_clock.cc:648] HybridClock initialized: now 1790513190457689 us; error 32 us; skew 500 ppm
I20260927 12:46:30.459645 5628 webserver.cc:460] Webserver started at http://127.0.98.62:40093/ using document root <none> and password file <none>
I20260927 12:46:30.460199 5628 fs_manager.cc:359] Metadata directory not provided
I20260927 12:46:30.460265 5628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:46:30.460448 5628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:46:30.462810 5628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790513161160093-392-0/minicluster-data/master-0/data/instance:
uuid: "5af343a4f9964e2688fff9c1e24221cc"
format_stamp: "Formatted at 2026-09-27 12:46:30 on dist-test-slave-lkn4"
server_key: "bea290d65444710a399fd2399fcb1d34"
server_key_iv: "61ac88cdb3d84860568599a97ed0d16f"
server_key_version: "encryptionkey@0"
I20260927 12:46:30.463382 5628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790513161160093-392-0/minicluster-data/master-0/wal/instance:
uuid: "5af343a4f9964e2688fff9c1e24221cc"
format_stamp: "Formatted at 2026-09-27 12:46:30 on dist-test-slave-lkn4"
server_key: "bea290d65444710a399fd2399fcb1d34"
server_key_iv: "61ac88cdb3d84860568599a97ed0d16f"
server_key_version: "encryptionkey@0"
I20260927 12:46:30.467139 5628 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 12:46:30.469493 5642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:30.470742 5628 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 12:46:30.470903 5628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790513161160093-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790513161160093-392-0/minicluster-data/master-0/wal
uuid: "5af343a4f9964e2688fff9c1e24221cc"
format_stamp: "Formatted at 2026-09-27 12:46:30 on dist-test-slave-lkn4"
server_key: "bea290d65444710a399fd2399fcb1d34"
server_key_iv: "61ac88cdb3d84860568599a97ed0d16f"
server_key_version: "encryptionkey@0"
I20260927 12:46:30.471004 5628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790513161160093-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790513161160093-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790513161160093-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:46:30.484658 5628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:46:30.485356 5628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:46:30.485527 5628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:46:30.491926 5628 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40773
I20260927 12:46:30.491955 5694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40773 every 8 connection(s)
I20260927 12:46:30.492842 5628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:46:30.495906 5695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:46:30.498678 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 5628
I20260927 12:46:30.498790 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790513161160093-392-0/minicluster-data/master-0/wal/instance
I20260927 12:46:30.499050 392 external_mini_cluster.cc:1428] Setting key 9488bafc7e6e5b2013b5f813b5e1371e
I20260927 12:46:30.501737 5695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5af343a4f9964e2688fff9c1e24221cc: Bootstrap starting.
I20260927 12:46:30.504361 5695 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5af343a4f9964e2688fff9c1e24221cc: Neither blocks nor log segments found. Creating new log.
I20260927 12:46:30.505162 5695 log.cc:824] T 00000000000000000000000000000000 P 5af343a4f9964e2688fff9c1e24221cc: Log is configured to *not* fsync() on all Append() calls
I20260927 12:46:30.507313 5695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5af343a4f9964e2688fff9c1e24221cc: No bootstrap required, opened a new log
I20260927 12:46:30.509346 5698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:30.500445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60528 (local address 127.0.98.62:40773)
0927 12:46:30.501083 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:30.501103 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:30.501145 (+ 42us) server_negotiation.cc:407] Connection header received
0927 12:46:30.501939 (+ 794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:30.501962 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:30.502467 (+ 505us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:30.502844 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:30.503299 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:30.504741 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:30.505691 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:30.506391 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:30.506548 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:30.507279 (+ 731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:30.507285 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:30.507301 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:46:30.507794 (+ 493us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:30.507864 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:46:30.507891 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:30.507986 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:30.508110 (+ 124us) server_negotiation.cc:299] Negotiation successful
0927 12:46:30.508394 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":96,"threads_started":1}
I20260927 12:46:30.511646 5695 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5af343a4f9964e2688fff9c1e24221cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5af343a4f9964e2688fff9c1e24221cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40773 } }
I20260927 12:46:30.511920 5695 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5af343a4f9964e2688fff9c1e24221cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:46:30.511979 5695 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5af343a4f9964e2688fff9c1e24221cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5af343a4f9964e2688fff9c1e24221cc, State: Initialized, Role: FOLLOWER
I20260927 12:46:30.512315 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:46:30.512435 5695 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5af343a4f9964e2688fff9c1e24221cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5af343a4f9964e2688fff9c1e24221cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40773 } }
I20260927 12:46:30.512570 5695 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5af343a4f9964e2688fff9c1e24221cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:46:30.512630 5695 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5af343a4f9964e2688fff9c1e24221cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:46:30.512730 5695 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5af343a4f9964e2688fff9c1e24221cc [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:46:30.513855 5695 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5af343a4f9964e2688fff9c1e24221cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5af343a4f9964e2688fff9c1e24221cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40773 } }
I20260927 12:46:30.514221 5695 leader_election.cc:302] T 00000000000000000000000000000000 P 5af343a4f9964e2688fff9c1e24221cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5af343a4f9964e2688fff9c1e24221cc; no voters:
I20260927 12:46:30.514497 5695 leader_election.cc:288] T 00000000000000000000000000000000 P 5af343a4f9964e2688fff9c1e24221cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:46:30.514613 5700 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5af343a4f9964e2688fff9c1e24221cc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:46:30.514823 5700 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5af343a4f9964e2688fff9c1e24221cc [term 1 LEADER]: Becoming Leader. State: Replica: 5af343a4f9964e2688fff9c1e24221cc, State: Running, Role: LEADER
I20260927 12:46:30.515228 5700 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5af343a4f9964e2688fff9c1e24221cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5af343a4f9964e2688fff9c1e24221cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40773 } }
I20260927 12:46:30.515726 5695 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5af343a4f9964e2688fff9c1e24221cc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:46:30.516700 5702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5af343a4f9964e2688fff9c1e24221cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5af343a4f9964e2688fff9c1e24221cc. Latest consensus state: current_term: 1 leader_uuid: "5af343a4f9964e2688fff9c1e24221cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5af343a4f9964e2688fff9c1e24221cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40773 } } }
I20260927 12:46:30.516820 5702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5af343a4f9964e2688fff9c1e24221cc [sys.catalog]: This master's current role is: LEADER
I20260927 12:46:30.517086 5701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5af343a4f9964e2688fff9c1e24221cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5af343a4f9964e2688fff9c1e24221cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5af343a4f9964e2688fff9c1e24221cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40773 } } }
I20260927 12:46:30.517204 5701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5af343a4f9964e2688fff9c1e24221cc [sys.catalog]: This master's current role is: LEADER
I20260927 12:46:30.517176 5709 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:46:30.520428 5709 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:46:30.525578 5709 catalog_manager.cc:1347] Generated new cluster ID: a770ae0ce49945d68b2ac37e3d7617b2
I20260927 12:46:30.525661 5709 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:46:30.546401 5709 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:46:30.547770 5709 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:46:30.567412 5709 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5af343a4f9964e2688fff9c1e24221cc: Generated new TSK 0
I20260927 12:46:30.568346 5709 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 12:46:30.600963 5725 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260927 12:46:30.601047 5725 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 12:46:30.601796 5698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:30.585599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60540 (local address 127.0.98.62:40773)
0927 12:46:30.585734 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:30.585739 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:30.585895 (+ 156us) server_negotiation.cc:407] Connection header received
0927 12:46:30.586077 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:30.586081 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:30.586140 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:30.586266 (+ 126us) server_negotiation.cc:226] Negotiated authn=JWT
0927 12:46:30.586467 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:30.587285 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:30.588151 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:30.588504 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:30.588604 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:30.588872 (+ 268us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 12:46:30.601379 (+ 12507us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36739/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 12:46:30.601644 (+ 265us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60540: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36739/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":50}
W20260927 12:46:30.601866 5698 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60540: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36739/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260927 12:46:30.601924 5724 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:46:30.585491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40773 (local address 127.0.0.1:60540)
0927 12:46:30.585808 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:30.585824 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:30.585932 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:30.586249 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:30.586251 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:30.586257 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0927 12:46:30.586348 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:30.586352 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:30.587420 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:30.587423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:30.588052 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:30.588060 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:30.588593 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:30.588610 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:30.588674 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:30.588700 (+ 26us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 12:46:30.601592 (+ 12892us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36739/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 12:46:30.601759 (+ 167us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:40773: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36739/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":111,"threads_started":1}
I20260927 12:46:30.602803 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 5628
2026-09-27T12:46:30Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (328 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260927 12:46:30.641718 392 mini_oidc.cc:152] Starting JWKS server
I20260927 12:46:30.642724 392 webserver.cc:460] Webserver started at https://127.0.0.1:45773/ using document root <none> and password file <none>
I20260927 12:46:30.642845 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 12:46:30.643136 392 webserver.cc:460] Webserver started at http://127.0.0.1:37741/ using document root <none> and password file <none>
2026-09-27T12:46:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:46:30Z Disabled control of system clock
I20260927 12:46:30.651757 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790513161160093-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790513161160093-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790513161160093-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38685
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38685
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45773/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 12:46:30.766198 5731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:46:30.766501 5731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:46:30.766541 5731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:46:30.770758 5731 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:46:30.770839 5731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:46:30.770853 5731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:46:30.770869 5731 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:46:30.770884 5731 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 12:46:30.770900 5731 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 12:46:30.770925 5731 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45773/jwks.json
I20260927 12:46:30.775972 5731 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790513161160093-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790513161160093-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38685
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45773/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790513161160093-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:46:30.777139 5731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:46:30.778585 5731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:46:30.787315 5736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:30.787319 5739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:30.787319 5737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:30.787375 5731 server_base.cc:1034] running on GCE node
I20260927 12:46:30.788156 5731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:46:30.789336 5731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:46:30.790494 5731 hybrid_clock.cc:648] HybridClock initialized: now 1790513190790472 us; error 44 us; skew 500 ppm
I20260927 12:46:30.792407 5731 webserver.cc:460] Webserver started at http://127.0.98.62:44853/ using document root <none> and password file <none>
I20260927 12:46:30.792972 5731 fs_manager.cc:359] Metadata directory not provided
I20260927 12:46:30.793040 5731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:46:30.793238 5731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:46:30.795379 5731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790513161160093-392-0/minicluster-data/master-0/data/instance:
uuid: "2fc63e4dec9646c28febda15227812de"
format_stamp: "Formatted at 2026-09-27 12:46:30 on dist-test-slave-lkn4"
server_key: "bca50129b9b1b4319c61b7dadcdc8c74"
server_key_iv: "cb68f38729801ef2a4bc11048dc844e7"
server_key_version: "encryptionkey@0"
I20260927 12:46:30.795964 5731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790513161160093-392-0/minicluster-data/master-0/wal/instance:
uuid: "2fc63e4dec9646c28febda15227812de"
format_stamp: "Formatted at 2026-09-27 12:46:30 on dist-test-slave-lkn4"
server_key: "bca50129b9b1b4319c61b7dadcdc8c74"
server_key_iv: "cb68f38729801ef2a4bc11048dc844e7"
server_key_version: "encryptionkey@0"
I20260927 12:46:30.799741 5731 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260927 12:46:30.802239 5745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:30.803697 5731 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 12:46:30.803834 5731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790513161160093-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790513161160093-392-0/minicluster-data/master-0/wal
uuid: "2fc63e4dec9646c28febda15227812de"
format_stamp: "Formatted at 2026-09-27 12:46:30 on dist-test-slave-lkn4"
server_key: "bca50129b9b1b4319c61b7dadcdc8c74"
server_key_iv: "cb68f38729801ef2a4bc11048dc844e7"
server_key_version: "encryptionkey@0"
I20260927 12:46:30.803943 5731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790513161160093-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790513161160093-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790513161160093-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:46:30.818920 5731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:46:30.819644 5731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:46:30.819819 5731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:46:30.826296 5731 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38685
I20260927 12:46:30.826316 5797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38685 every 8 connection(s)
I20260927 12:46:30.827373 5731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:46:30.827837 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 5731
I20260927 12:46:30.827952 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790513161160093-392-0/minicluster-data/master-0/wal/instance
I20260927 12:46:30.828236 392 external_mini_cluster.cc:1428] Setting key 968f2b03939b9e1bb64b9df0f6f6a65e
I20260927 12:46:30.830827 5798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:46:30.836537 5798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fc63e4dec9646c28febda15227812de: Bootstrap starting.
I20260927 12:46:30.837632 5801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:30.829623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57636 (local address 127.0.98.62:38685)
0927 12:46:30.830390 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:30.830412 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:30.830462 (+ 50us) server_negotiation.cc:407] Connection header received
0927 12:46:30.831199 (+ 737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:30.831217 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:30.831661 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:30.832015 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:30.832465 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:30.833466 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:30.834400 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:30.834901 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:30.835048 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:30.835892 (+ 844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:30.835897 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:30.835910 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:46:30.836295 (+ 385us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:30.836353 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:46:30.836372 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:30.836459 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:30.836602 (+ 143us) server_negotiation.cc:299] Negotiation successful
0927 12:46:30.836904 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":123,"threads_started":1}
I20260927 12:46:30.838822 5798 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2fc63e4dec9646c28febda15227812de: Neither blocks nor log segments found. Creating new log.
I20260927 12:46:30.839533 5798 log.cc:824] T 00000000000000000000000000000000 P 2fc63e4dec9646c28febda15227812de: Log is configured to *not* fsync() on all Append() calls
I20260927 12:46:30.840454 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:46:30.841624 5798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fc63e4dec9646c28febda15227812de: No bootstrap required, opened a new log
I20260927 12:46:30.845158 5798 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2fc63e4dec9646c28febda15227812de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fc63e4dec9646c28febda15227812de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38685 } }
I20260927 12:46:30.845396 5798 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2fc63e4dec9646c28febda15227812de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:46:30.845453 5798 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2fc63e4dec9646c28febda15227812de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fc63e4dec9646c28febda15227812de, State: Initialized, Role: FOLLOWER
I20260927 12:46:30.846076 5798 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2fc63e4dec9646c28febda15227812de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fc63e4dec9646c28febda15227812de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38685 } }
I20260927 12:46:30.846235 5798 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2fc63e4dec9646c28febda15227812de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:46:30.846304 5798 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2fc63e4dec9646c28febda15227812de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:46:30.846433 5798 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2fc63e4dec9646c28febda15227812de [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:46:30.847422 5798 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2fc63e4dec9646c28febda15227812de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fc63e4dec9646c28febda15227812de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38685 } }
I20260927 12:46:30.847792 5798 leader_election.cc:302] T 00000000000000000000000000000000 P 2fc63e4dec9646c28febda15227812de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2fc63e4dec9646c28febda15227812de; no voters:
I20260927 12:46:30.848169 5798 leader_election.cc:288] T 00000000000000000000000000000000 P 2fc63e4dec9646c28febda15227812de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:46:30.848264 5803 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2fc63e4dec9646c28febda15227812de [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:46:30.848449 5803 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2fc63e4dec9646c28febda15227812de [term 1 LEADER]: Becoming Leader. State: Replica: 2fc63e4dec9646c28febda15227812de, State: Running, Role: LEADER
I20260927 12:46:30.848829 5803 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2fc63e4dec9646c28febda15227812de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fc63e4dec9646c28febda15227812de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38685 } }
I20260927 12:46:30.849572 5798 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2fc63e4dec9646c28febda15227812de [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:46:30.850675 5804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fc63e4dec9646c28febda15227812de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2fc63e4dec9646c28febda15227812de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fc63e4dec9646c28febda15227812de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38685 } } }
I20260927 12:46:30.850651 5805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fc63e4dec9646c28febda15227812de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2fc63e4dec9646c28febda15227812de. Latest consensus state: current_term: 1 leader_uuid: "2fc63e4dec9646c28febda15227812de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fc63e4dec9646c28febda15227812de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38685 } } }
I20260927 12:46:30.850798 5804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fc63e4dec9646c28febda15227812de [sys.catalog]: This master's current role is: LEADER
I20260927 12:46:30.850798 5805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fc63e4dec9646c28febda15227812de [sys.catalog]: This master's current role is: LEADER
I20260927 12:46:30.851248 5812 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:46:30.854710 5812 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:46:30.860718 5812 catalog_manager.cc:1347] Generated new cluster ID: b055874f626b446db3ed182bba624007
I20260927 12:46:30.860800 5812 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:46:30.870085 5812 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:46:30.871356 5812 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:46:30.884744 5812 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2fc63e4dec9646c28febda15227812de: Generated new TSK 0
I20260927 12:46:30.885483 5812 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:46:30.899962 5801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:30.894821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57646 (local address 127.0.98.62:38685)
0927 12:46:30.894951 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:30.894955 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:30.895119 (+ 164us) server_negotiation.cc:407] Connection header received
0927 12:46:30.895318 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:30.895321 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:30.895367 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:30.895455 (+ 88us) server_negotiation.cc:226] Negotiated authn=JWT
0927 12:46:30.895783 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:30.896317 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:30.897543 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:30.897828 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:30.897918 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:30.898286 (+ 368us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 12:46:30.899615 (+ 1329us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45773/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 12:46:30.899842 (+ 227us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57646: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45773/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":49}
W20260927 12:46:30.900031 5801 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57646: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45773/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260927 12:46:30.903093 5827 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:46:30.894743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38685 (local address 127.0.0.1:57646)
0927 12:46:30.895031 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:30.895046 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:30.895163 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:30.895473 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:30.895477 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:30.895485 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0927 12:46:30.895612 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:30.895619 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:30.896421 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:30.896425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:30.897412 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:30.897423 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:30.897929 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:30.897954 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:30.898070 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:30.898101 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 12:46:30.899779 (+ 1678us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45773/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 12:46:30.902942 (+ 3163us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:38685: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45773/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":2861,"thread_start_us":91,"threads_started":1}
I20260927 12:46:30.903858 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 5731
E20260927 12:46:30.914474 5828 webserver.cc:533] Webserver: sslize failed:
E20260927 12:46:30.914563 5828 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-27T12:46:30Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (302 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-27T12:46:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:46:30Z Disabled control of system clock
I20260927 12:46:30.952672 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790513161160093-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790513161160093-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790513161160093-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43853
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43853
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790513161160093-392-0/testchainca.cert with env {}
W20260927 12:46:31.063663 5832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:46:31.063966 5832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:46:31.064015 5832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:46:31.068176 5832 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:46:31.068254 5832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:46:31.068269 5832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:46:31.068285 5832 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:46:31.068300 5832 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 12:46:31.068315 5832 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 12:46:31.068346 5832 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260927 12:46:31.074106 5832 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790513161160093-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790513161160093-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43853
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790513161160093-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790513161160093-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:46:31.075222 5832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:46:31.076251 5832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 12:46:31.084743 5832 server_base.cc:1034] running on GCE node
W20260927 12:46:31.084654 5838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:31.084666 5840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:31.084652 5837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:31.085600 5832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:46:31.086792 5832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:46:31.087961 5832 hybrid_clock.cc:648] HybridClock initialized: now 1790513191087941 us; error 30 us; skew 500 ppm
I20260927 12:46:31.089913 5832 webserver.cc:460] Webserver started at http://127.0.98.62:45091/ using document root <none> and password file <none>
I20260927 12:46:31.090494 5832 fs_manager.cc:359] Metadata directory not provided
I20260927 12:46:31.090570 5832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:46:31.090758 5832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:46:31.092960 5832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790513161160093-392-0/minicluster-data/master-0/data/instance:
uuid: "dd61ca817f4142f98be1952165165671"
format_stamp: "Formatted at 2026-09-27 12:46:31 on dist-test-slave-lkn4"
server_key: "6ddc849e0c7592733f5aa6b19d7c6d8c"
server_key_iv: "d24a5ab14d0a741aa29c827c47a8a5eb"
server_key_version: "encryptionkey@0"
I20260927 12:46:31.093510 5832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790513161160093-392-0/minicluster-data/master-0/wal/instance:
uuid: "dd61ca817f4142f98be1952165165671"
format_stamp: "Formatted at 2026-09-27 12:46:31 on dist-test-slave-lkn4"
server_key: "6ddc849e0c7592733f5aa6b19d7c6d8c"
server_key_iv: "d24a5ab14d0a741aa29c827c47a8a5eb"
server_key_version: "encryptionkey@0"
I20260927 12:46:31.097536 5832 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 12:46:31.100293 5846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:31.101828 5832 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 12:46:31.101955 5832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790513161160093-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790513161160093-392-0/minicluster-data/master-0/wal
uuid: "dd61ca817f4142f98be1952165165671"
format_stamp: "Formatted at 2026-09-27 12:46:31 on dist-test-slave-lkn4"
server_key: "6ddc849e0c7592733f5aa6b19d7c6d8c"
server_key_iv: "d24a5ab14d0a741aa29c827c47a8a5eb"
server_key_version: "encryptionkey@0"
I20260927 12:46:31.102113 5832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790513161160093-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790513161160093-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790513161160093-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:46:31.120847 5832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:46:31.121592 5832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:46:31.121789 5832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:46:31.128791 5832 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43853
I20260927 12:46:31.128808 5898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43853 every 8 connection(s)
I20260927 12:46:31.129863 5832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:46:31.133136 5899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:46:31.138671 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 5832
I20260927 12:46:31.138844 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790513161160093-392-0/minicluster-data/master-0/wal/instance
I20260927 12:46:31.139289 392 external_mini_cluster.cc:1428] Setting key 47f6aeb4265fb85915708c9bb75647a6
I20260927 12:46:31.139645 5899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd61ca817f4142f98be1952165165671: Bootstrap starting.
I20260927 12:46:31.142544 5899 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd61ca817f4142f98be1952165165671: Neither blocks nor log segments found. Creating new log.
I20260927 12:46:31.143396 5899 log.cc:824] T 00000000000000000000000000000000 P dd61ca817f4142f98be1952165165671: Log is configured to *not* fsync() on all Append() calls
I20260927 12:46:31.145570 5899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd61ca817f4142f98be1952165165671: No bootstrap required, opened a new log
I20260927 12:46:31.148890 5902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:31.140707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46618 (local address 127.0.98.62:43853)
0927 12:46:31.141429 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:31.141443 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:31.141482 (+ 39us) server_negotiation.cc:407] Connection header received
0927 12:46:31.142226 (+ 744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:31.142245 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:31.142703 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:31.143029 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:31.143435 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:31.144424 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:31.145558 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:31.146104 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:31.146221 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:31.147031 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:31.147037 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:31.147051 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:46:31.147495 (+ 444us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:31.147555 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:46:31.147576 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:31.147649 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:31.147830 (+ 181us) server_negotiation.cc:299] Negotiation successful
0927 12:46:31.148117 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":150,"threads_started":1}
I20260927 12:46:31.149705 5899 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd61ca817f4142f98be1952165165671 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd61ca817f4142f98be1952165165671" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43853 } }
I20260927 12:46:31.149932 5899 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd61ca817f4142f98be1952165165671 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:46:31.149989 5899 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd61ca817f4142f98be1952165165671 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd61ca817f4142f98be1952165165671, State: Initialized, Role: FOLLOWER
I20260927 12:46:31.150413 5899 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd61ca817f4142f98be1952165165671 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd61ca817f4142f98be1952165165671" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43853 } }
I20260927 12:46:31.150539 5899 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd61ca817f4142f98be1952165165671 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:46:31.150636 5899 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd61ca817f4142f98be1952165165671 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:46:31.150751 5899 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd61ca817f4142f98be1952165165671 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:46:31.151707 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:46:31.151813 5899 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd61ca817f4142f98be1952165165671 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd61ca817f4142f98be1952165165671" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43853 } }
I20260927 12:46:31.152125 5899 leader_election.cc:302] T 00000000000000000000000000000000 P dd61ca817f4142f98be1952165165671 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd61ca817f4142f98be1952165165671; no voters:
I20260927 12:46:31.152417 5899 leader_election.cc:288] T 00000000000000000000000000000000 P dd61ca817f4142f98be1952165165671 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:46:31.152527 5904 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd61ca817f4142f98be1952165165671 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:46:31.152736 5904 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd61ca817f4142f98be1952165165671 [term 1 LEADER]: Becoming Leader. State: Replica: dd61ca817f4142f98be1952165165671, State: Running, Role: LEADER
I20260927 12:46:31.153070 5904 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd61ca817f4142f98be1952165165671 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd61ca817f4142f98be1952165165671" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43853 } }
I20260927 12:46:31.153739 5899 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd61ca817f4142f98be1952165165671 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:46:31.154206 5906 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd61ca817f4142f98be1952165165671 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd61ca817f4142f98be1952165165671. Latest consensus state: current_term: 1 leader_uuid: "dd61ca817f4142f98be1952165165671" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd61ca817f4142f98be1952165165671" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43853 } } }
I20260927 12:46:31.154384 5906 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd61ca817f4142f98be1952165165671 [sys.catalog]: This master's current role is: LEADER
I20260927 12:46:31.154534 5905 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd61ca817f4142f98be1952165165671 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd61ca817f4142f98be1952165165671" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd61ca817f4142f98be1952165165671" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43853 } } }
I20260927 12:46:31.154685 5905 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd61ca817f4142f98be1952165165671 [sys.catalog]: This master's current role is: LEADER
I20260927 12:46:31.155129 5911 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:46:31.158200 5911 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:46:31.163463 5911 catalog_manager.cc:1347] Generated new cluster ID: 5424aa81f7ba43ac8a6745a3b0f8ec25
I20260927 12:46:31.163555 5911 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:46:31.177652 5911 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:46:31.178704 5911 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:46:31.188261 5911 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd61ca817f4142f98be1952165165671: Generated new TSK 0
I20260927 12:46:31.189054 5911 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:46:31.255215 5902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:31.198177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46630 (local address 127.0.98.62:43853)
0927 12:46:31.198308 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:31.198313 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:31.198459 (+ 146us) server_negotiation.cc:407] Connection header received
0927 12:46:31.198778 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:31.198781 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:31.198839 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:31.198942 (+ 103us) server_negotiation.cc:226] Negotiated authn=JWT
0927 12:46:31.199215 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:31.199935 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:31.201208 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:31.201433 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:31.201521 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:31.201888 (+ 367us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 12:46:31.254634 (+ 52746us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 12:46:31.255094 (+ 460us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46630: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":57}
W20260927 12:46:31.255288 5902 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46630: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260927 12:46:31.255426 5928 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:46:31.198059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43853 (local address 127.0.0.1:46630)
0927 12:46:31.198362 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:31.198378 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:31.198506 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:31.198958 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:31.198961 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:31.198969 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0927 12:46:31.199084 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:31.199090 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:31.200058 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:31.200063 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:31.201073 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:31.201085 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:31.201526 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:31.201550 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:31.201647 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:31.201678 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 12:46:31.254947 (+ 53269us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 12:46:31.255233 (+ 286us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:43853: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":110,"threads_started":1}
I20260927 12:46:31.256402 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 5832
2026-09-27T12:46:31Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (330 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (121 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (121 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:46:31 dist-test-slave-lkn4 krb5kdc[5940](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:46:31 dist-test-slave-lkn4 krb5kdc[5940](info): set up 2 sockets
Sep 27 12:46:31 dist-test-slave-lkn4 krb5kdc[5940](info): commencing operation
krb5kdc: starting...
W20260927 12:46:32.454236 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/krb5kdc/test-user.keytab.
Sep 27 12:46:32 dist-test-slave-lkn4 krb5kdc[5940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513192, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:46:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:46:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:46:32.575205 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45241
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45241
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:46:32.685019 5953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:46:32.685396 5953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:46:32.685473 5953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:46:32.691443 5953 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:46:32.691530 5953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:46:32.691560 5953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:46:32.691581 5953 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:46:32.691608 5953 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:46:32.697496 5953 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45241
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:46:32.698681 5953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:46:32.699692 5953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:46:32.708045 5961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:32.708132 5953 server_base.cc:1034] running on GCE node
W20260927 12:46:32.708045 5958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:32.708055 5959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:32.708909 5953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:46:32.710108 5953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:46:32.711304 5953 hybrid_clock.cc:648] HybridClock initialized: now 1790513192711277 us; error 43 us; skew 500 ppm
Sep 27 12:46:32 dist-test-slave-lkn4 krb5kdc[5940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513192, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:46:32.714564 5953 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:46:32.715487 5953 webserver.cc:460] Webserver started at http://127.0.98.62:36077/ using document root <none> and password file <none>
I20260927 12:46:32.716037 5953 fs_manager.cc:359] Metadata directory not provided
I20260927 12:46:32.716100 5953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:46:32.716321 5953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:46:32.718506 5953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/minicluster-data/master-0/data/instance:
uuid: "01d9671926114fceb705ce3a4fbad23e"
format_stamp: "Formatted at 2026-09-27 12:46:32 on dist-test-slave-lkn4"
server_key: "7a6bb4c1f73c89fcd1358c975a931675"
server_key_iv: "abb3115b92b67b0524b734257420a1e2"
server_key_version: "encryptionkey@0"
I20260927 12:46:32.719055 5953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/minicluster-data/master-0/wal/instance:
uuid: "01d9671926114fceb705ce3a4fbad23e"
format_stamp: "Formatted at 2026-09-27 12:46:32 on dist-test-slave-lkn4"
server_key: "7a6bb4c1f73c89fcd1358c975a931675"
server_key_iv: "abb3115b92b67b0524b734257420a1e2"
server_key_version: "encryptionkey@0"
I20260927 12:46:32.722973 5953 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 12:46:32.725471 5968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:32.726914 5953 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:46:32.727028 5953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/minicluster-data/master-0/wal
uuid: "01d9671926114fceb705ce3a4fbad23e"
format_stamp: "Formatted at 2026-09-27 12:46:32 on dist-test-slave-lkn4"
server_key: "7a6bb4c1f73c89fcd1358c975a931675"
server_key_iv: "abb3115b92b67b0524b734257420a1e2"
server_key_version: "encryptionkey@0"
I20260927 12:46:32.727142 5953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:46:32.742874 5953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:46:32.746068 5953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:46:32.746253 5953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:46:32.753120 5953 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45241
I20260927 12:46:32.753404 6020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45241 every 8 connection(s)
I20260927 12:46:32.754161 5953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:46:32.756956 6021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:46:32.761361 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 5953
I20260927 12:46:32.761492 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/minicluster-data/master-0/wal/instance
I20260927 12:46:32.761818 392 external_mini_cluster.cc:1428] Setting key 50419eebdd16a3d6fb1fa6bd70b93c5f
I20260927 12:46:32.762792 6021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01d9671926114fceb705ce3a4fbad23e: Bootstrap starting.
I20260927 12:46:32.765103 6021 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01d9671926114fceb705ce3a4fbad23e: Neither blocks nor log segments found. Creating new log.
I20260927 12:46:32.766028 6021 log.cc:824] T 00000000000000000000000000000000 P 01d9671926114fceb705ce3a4fbad23e: Log is configured to *not* fsync() on all Append() calls
I20260927 12:46:32.768051 6021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01d9671926114fceb705ce3a4fbad23e: No bootstrap required, opened a new log
Sep 27 12:46:32 dist-test-slave-lkn4 krb5kdc[5940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513192, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:46:32.772209 6021 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01d9671926114fceb705ce3a4fbad23e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01d9671926114fceb705ce3a4fbad23e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45241 } }
I20260927 12:46:32.772492 6021 raft_consensus.cc:374] T 00000000000000000000000000000000 P 01d9671926114fceb705ce3a4fbad23e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:46:32.772537 6021 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01d9671926114fceb705ce3a4fbad23e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01d9671926114fceb705ce3a4fbad23e, State: Initialized, Role: FOLLOWER
I20260927 12:46:32.773020 6021 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01d9671926114fceb705ce3a4fbad23e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01d9671926114fceb705ce3a4fbad23e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45241 } }
I20260927 12:46:32.773146 6021 raft_consensus.cc:388] T 00000000000000000000000000000000 P 01d9671926114fceb705ce3a4fbad23e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:46:32.773217 6021 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01d9671926114fceb705ce3a4fbad23e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:46:32.773312 6021 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01d9671926114fceb705ce3a4fbad23e [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:46:32.774340 6021 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01d9671926114fceb705ce3a4fbad23e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01d9671926114fceb705ce3a4fbad23e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45241 } }
I20260927 12:46:32.774698 6021 leader_election.cc:302] T 00000000000000000000000000000000 P 01d9671926114fceb705ce3a4fbad23e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01d9671926114fceb705ce3a4fbad23e; no voters:
I20260927 12:46:32.774991 6021 leader_election.cc:288] T 00000000000000000000000000000000 P 01d9671926114fceb705ce3a4fbad23e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:46:32.775070 6026 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01d9671926114fceb705ce3a4fbad23e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:46:32.775236 6026 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01d9671926114fceb705ce3a4fbad23e [term 1 LEADER]: Becoming Leader. State: Replica: 01d9671926114fceb705ce3a4fbad23e, State: Running, Role: LEADER
I20260927 12:46:32.775581 6026 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01d9671926114fceb705ce3a4fbad23e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01d9671926114fceb705ce3a4fbad23e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45241 } }
I20260927 12:46:32.776294 6021 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01d9671926114fceb705ce3a4fbad23e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:46:32.776826 6027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01d9671926114fceb705ce3a4fbad23e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01d9671926114fceb705ce3a4fbad23e. Latest consensus state: current_term: 1 leader_uuid: "01d9671926114fceb705ce3a4fbad23e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01d9671926114fceb705ce3a4fbad23e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45241 } } }
I20260927 12:46:32.776932 6027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01d9671926114fceb705ce3a4fbad23e [sys.catalog]: This master's current role is: LEADER
I20260927 12:46:32.776988 6024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:32.763380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52058 (local address 127.0.98.62:45241)
0927 12:46:32.763833 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:32.763848 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:32.763886 (+ 38us) server_negotiation.cc:407] Connection header received
0927 12:46:32.764530 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:32.764547 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:32.764916 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:32.765235 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:32.766122 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:32.767167 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:32.768262 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:32.768732 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:32.768858 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:32.771798 (+ 2940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:32.771803 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:32.771818 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:32.771845 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:32.773717 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:32.774219 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:32.774225 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:32.774230 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:32.774358 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:32.774634 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:32.774638 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:32.774640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:32.775068 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:32.775319 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:32.775593 (+ 274us) server_negotiation.cc:299] Negotiation successful
0927 12:46:32.776111 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":299,"thread_start_us":132,"threads_started":1}
I20260927 12:46:32.777796 6028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01d9671926114fceb705ce3a4fbad23e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "01d9671926114fceb705ce3a4fbad23e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01d9671926114fceb705ce3a4fbad23e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45241 } } }
I20260927 12:46:32.777889 6028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01d9671926114fceb705ce3a4fbad23e [sys.catalog]: This master's current role is: LEADER
W20260927 12:46:32.781217 6041 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 01d9671926114fceb705ce3a4fbad23e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 12:46:32.781286 6041 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 12:46:32.781355 6031 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:46:32.782119 6031 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:46:32.788506 6031 catalog_manager.cc:1347] Generated new cluster ID: b824a5d4f6c346a696cabaa9f106feef
I20260927 12:46:32.788586 6031 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:46:32.798529 6031 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:46:32.799824 6031 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:46:32.806981 6031 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 01d9671926114fceb705ce3a4fbad23e: Generated new TSK 0
I20260927 12:46:32.807753 6031 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 12:46:32.841003 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45241
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:46:32.956549 6047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:46:32.956856 6047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:46:32.956900 6047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:46:32.961092 6047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:46:32.961170 6047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:46:32.961239 6047 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:46:32.966681 6047 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6047
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:46:32.967828 6047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:46:32.968894 6047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:46:32.978339 6052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:32.978403 6055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:32.978396 6053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:32.978685 6047 server_base.cc:1034] running on GCE node
I20260927 12:46:32.979213 6047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:46:32.979946 6047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:46:32.981087 6047 hybrid_clock.cc:648] HybridClock initialized: now 1790513192981047 us; error 61 us; skew 500 ppm
Sep 27 12:46:32 dist-test-slave-lkn4 krb5kdc[5940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513192, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:46:32.983958 6047 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 12:46:32.984896 6047 webserver.cc:460] Webserver started at http://127.0.98.1:35485/ using document root <none> and password file <none>
I20260927 12:46:32.985455 6047 fs_manager.cc:359] Metadata directory not provided
I20260927 12:46:32.985515 6047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:46:32.985719 6047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:46:32.987870 6047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/minicluster-data/ts-0/data/instance:
uuid: "e29023dd57064b43bb505eecca02dc69"
format_stamp: "Formatted at 2026-09-27 12:46:32 on dist-test-slave-lkn4"
server_key: "e2c84bba54b3aa5183dc7558cbfb88d8"
server_key_iv: "f8750ea5ab9feba8780b3d900269ce35"
server_key_version: "encryptionkey@0"
I20260927 12:46:32.988420 6047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e29023dd57064b43bb505eecca02dc69"
format_stamp: "Formatted at 2026-09-27 12:46:32 on dist-test-slave-lkn4"
server_key: "e2c84bba54b3aa5183dc7558cbfb88d8"
server_key_iv: "f8750ea5ab9feba8780b3d900269ce35"
server_key_version: "encryptionkey@0"
I20260927 12:46:32.992172 6047 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 12:46:32.994771 6062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:32.996090 6047 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:46:32.996223 6047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/minicluster-data/ts-0/wal
uuid: "e29023dd57064b43bb505eecca02dc69"
format_stamp: "Formatted at 2026-09-27 12:46:32 on dist-test-slave-lkn4"
server_key: "e2c84bba54b3aa5183dc7558cbfb88d8"
server_key_iv: "f8750ea5ab9feba8780b3d900269ce35"
server_key_version: "encryptionkey@0"
I20260927 12:46:32.996330 6047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:46:33.016831 6047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:46:33.020666 6047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:46:33.020965 6047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:46:33.021715 6047 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:46:33.022655 6047 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:46:33.022702 6047 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:33.022743 6047 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:46:33.022758 6047 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:33.031716 6047 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39903
I20260927 12:46:33.031821 6175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39903 every 8 connection(s)
I20260927 12:46:33.032706 6047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:46:33.038607 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 6047
I20260927 12:46:33.038729 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:46:33.038987 392 external_mini_cluster.cc:1428] Setting key c8e261907e99807ba9f65f72e1d1a2f2
Sep 27 12:46:33 dist-test-slave-lkn4 krb5kdc[5940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513192, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:46:33.046548 6024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:33.034624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33797 (local address 127.0.98.62:45241)
0927 12:46:33.034766 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:33.034770 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:33.035659 (+ 889us) server_negotiation.cc:407] Connection header received
0927 12:46:33.037065 (+ 1406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:33.037068 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:33.037127 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:33.037202 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:33.038447 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:33.039089 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:33.040280 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:33.040588 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:33.040667 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:33.043206 (+ 2539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:33.043210 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:33.043213 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:33.043237 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:33.044602 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:33.045068 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:33.045072 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:33.045074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:33.045148 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:33.045448 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:33.045452 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:33.045454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:33.045636 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:33.045725 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:33.046250 (+ 525us) server_negotiation.cc:299] Negotiation successful
0927 12:46:33.046381 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 12:46:33.047386 6178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:33.035034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45241 (local address 127.0.98.1:33797)
0927 12:46:33.035497 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:33.035547 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:33.036849 (+ 1302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:33.037386 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:33.037394 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:33.037757 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:33.038304 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:33.038318 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:33.039215 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:33.039218 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:33.040163 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:33.040172 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:33.040735 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:33.040740 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:33.040928 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:33.041535 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:33.041555 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:33.043055 (+ 1500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:33.044706 (+ 1651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:33.044712 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:33.044729 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:33.044979 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:33.045230 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:33.045233 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:33.045235 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:33.045369 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:33.045729 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:33.045735 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:33.045983 (+ 248us) client_negotiation.cc:769] Sending connection context
0927 12:46:33.046189 (+ 206us) client_negotiation.cc:240] Negotiation successful
0927 12:46:33.046519 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":104,"threads_started":1}
I20260927 12:46:33.048595 6176 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45241
I20260927 12:46:33.048840 6176 heartbeater.cc:464] Registering TS with master...
I20260927 12:46:33.049350 6176 heartbeater.cc:511] Master 127.0.98.62:45241 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 12:46:33.050885 5985 ts_manager.cc:194] Registered new tserver with Master: e29023dd57064b43bb505eecca02dc69 (127.0.98.1:39903)
I20260927 12:46:33.052393 5985 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33797
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 12:46:33.065331 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45241
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:46:33.174218 6181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:46:33.174474 6181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:46:33.174531 6181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:46:33.178620 6181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:46:33.178689 6181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:46:33.178771 6181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:46:33.183980 6181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:46:33.185115 6181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:46:33.186168 6181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:46:33.195628 6189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:33.195755 6187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:33.195643 6186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:33.195938 6181 server_base.cc:1034] running on GCE node
I20260927 12:46:33.196333 6181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:46:33.196945 6181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:46:33.198124 6181 hybrid_clock.cc:648] HybridClock initialized: now 1790513193198103 us; error 27 us; skew 500 ppm
Sep 27 12:46:33 dist-test-slave-lkn4 krb5kdc[5940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513193, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:46:33.200841 6181 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 12:46:33.201659 6181 webserver.cc:460] Webserver started at http://127.0.98.2:35153/ using document root <none> and password file <none>
I20260927 12:46:33.202225 6181 fs_manager.cc:359] Metadata directory not provided
I20260927 12:46:33.202291 6181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:46:33.202481 6181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:46:33.204445 6181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/minicluster-data/ts-1/data/instance:
uuid: "f46d37b9a81842d5a2dd0bf62cc13c65"
format_stamp: "Formatted at 2026-09-27 12:46:33 on dist-test-slave-lkn4"
server_key: "f0978a5e134da86337b27368a0121d47"
server_key_iv: "990ccf6d89aa82887d90bb15007f21e7"
server_key_version: "encryptionkey@0"
I20260927 12:46:33.204950 6181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f46d37b9a81842d5a2dd0bf62cc13c65"
format_stamp: "Formatted at 2026-09-27 12:46:33 on dist-test-slave-lkn4"
server_key: "f0978a5e134da86337b27368a0121d47"
server_key_iv: "990ccf6d89aa82887d90bb15007f21e7"
server_key_version: "encryptionkey@0"
I20260927 12:46:33.208532 6181 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 12:46:33.211088 6196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:33.212312 6181 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:46:33.212945 6181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/minicluster-data/ts-1/wal
uuid: "f46d37b9a81842d5a2dd0bf62cc13c65"
format_stamp: "Formatted at 2026-09-27 12:46:33 on dist-test-slave-lkn4"
server_key: "f0978a5e134da86337b27368a0121d47"
server_key_iv: "990ccf6d89aa82887d90bb15007f21e7"
server_key_version: "encryptionkey@0"
I20260927 12:46:33.213048 6181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:46:33.241840 6181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:46:33.245012 6181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:46:33.245223 6181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:46:33.245869 6181 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:46:33.246815 6181 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:46:33.246868 6181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:33.246927 6181 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:46:33.246958 6181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:33.256165 6181 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36681
I20260927 12:46:33.256193 6309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36681 every 8 connection(s)
I20260927 12:46:33.257102 6181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:46:33.261752 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 6181
I20260927 12:46:33.261864 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:46:33.262157 392 external_mini_cluster.cc:1428] Setting key dabda074396782491d9859428a38376d
Sep 27 12:46:33 dist-test-slave-lkn4 krb5kdc[5940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513193, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 12:46:33.271960 6024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:33.259101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42359 (local address 127.0.98.62:45241)
0927 12:46:33.259272 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:33.259276 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:33.260123 (+ 847us) server_negotiation.cc:407] Connection header received
0927 12:46:33.261446 (+ 1323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:33.261450 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:33.261506 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:33.261599 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:33.263149 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:33.263839 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:33.265491 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:33.265836 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:33.265944 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:33.268625 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:33.268629 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:33.268632 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:33.268657 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:33.269996 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:33.270502 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:33.270506 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:33.270508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:33.270582 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:33.270881 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:33.270884 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:33.270886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:33.271066 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:33.271147 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:33.271659 (+ 512us) server_negotiation.cc:299] Negotiation successful
0927 12:46:33.271793 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 12:46:33.272874 6312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:33.259475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45241 (local address 127.0.98.2:42359)
0927 12:46:33.259963 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:33.260006 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:33.261210 (+ 1204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:33.261865 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:33.261874 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:33.262356 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:33.262996 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:33.263014 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:33.263974 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:33.263977 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:33.265332 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:33.265344 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:33.265976 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:33.265979 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:33.266175 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:33.266817 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:33.266840 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:33.268454 (+ 1614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:33.270130 (+ 1676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:33.270137 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:33.270154 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:33.270411 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:33.270660 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:33.270663 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:33.270665 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:33.270799 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:33.271157 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:33.271164 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:33.271436 (+ 272us) client_negotiation.cc:769] Sending connection context
0927 12:46:33.271605 (+ 169us) client_negotiation.cc:240] Negotiation successful
0927 12:46:33.271933 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":123,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 12:46:33.274103 6310 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45241
I20260927 12:46:33.274387 6310 heartbeater.cc:464] Registering TS with master...
I20260927 12:46:33.275056 6310 heartbeater.cc:511] Master 127.0.98.62:45241 requested a full tablet report, sending...
I20260927 12:46:33.276732 5985 ts_manager.cc:194] Registered new tserver with Master: f46d37b9a81842d5a2dd0bf62cc13c65 (127.0.98.2:36681)
I20260927 12:46:33.277490 5985 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42359
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 12:46:33.289347 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45241
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:46:33.401597 6315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:46:33.401880 6315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:46:33.401940 6315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:46:33.405916 6315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:46:33.405980 6315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:46:33.406080 6315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:46:33.411337 6315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45241
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:46:33.412447 6315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:46:33.413501 6315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:46:33.423012 6320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:33.422982 6323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:33.423002 6321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:33.423302 6315 server_base.cc:1034] running on GCE node
I20260927 12:46:33.423718 6315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:46:33.424319 6315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:46:33.425477 6315 hybrid_clock.cc:648] HybridClock initialized: now 1790513193425445 us; error 44 us; skew 500 ppm
Sep 27 12:46:33 dist-test-slave-lkn4 krb5kdc[5940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513193, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:46:33.428354 6315 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 12:46:33.429255 6315 webserver.cc:460] Webserver started at http://127.0.98.3:41151/ using document root <none> and password file <none>
I20260927 12:46:33.429814 6315 fs_manager.cc:359] Metadata directory not provided
I20260927 12:46:33.429878 6315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:46:33.430133 6315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:46:33.432286 6315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/minicluster-data/ts-2/data/instance:
uuid: "6acdaeac2ecc4ebfae1e66311b7a3427"
format_stamp: "Formatted at 2026-09-27 12:46:33 on dist-test-slave-lkn4"
server_key: "cbb1260ef6bda9d3848f7c56bd0e63fb"
server_key_iv: "c197254d07083d32f68405cc7dc61076"
server_key_version: "encryptionkey@0"
I20260927 12:46:33.432864 6315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6acdaeac2ecc4ebfae1e66311b7a3427"
format_stamp: "Formatted at 2026-09-27 12:46:33 on dist-test-slave-lkn4"
server_key: "cbb1260ef6bda9d3848f7c56bd0e63fb"
server_key_iv: "c197254d07083d32f68405cc7dc61076"
server_key_version: "encryptionkey@0"
I20260927 12:46:33.436792 6315 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 12:46:33.439510 6330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:33.440872 6315 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 12:46:33.440992 6315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/minicluster-data/ts-2/wal
uuid: "6acdaeac2ecc4ebfae1e66311b7a3427"
format_stamp: "Formatted at 2026-09-27 12:46:33 on dist-test-slave-lkn4"
server_key: "cbb1260ef6bda9d3848f7c56bd0e63fb"
server_key_iv: "c197254d07083d32f68405cc7dc61076"
server_key_version: "encryptionkey@0"
I20260927 12:46:33.441098 6315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:46:33.454358 6315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:46:33.457886 6315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:46:33.458103 6315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:46:33.458731 6315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:46:33.459568 6315 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:46:33.459625 6315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:33.459685 6315 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:46:33.459709 6315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:33.468287 6315 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43893
I20260927 12:46:33.468321 6443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43893 every 8 connection(s)
I20260927 12:46:33.469224 6315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:46:33.475580 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 6315
I20260927 12:46:33.475684 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:46:33.475936 392 external_mini_cluster.cc:1428] Setting key e19b0c24dc9783f9aea5567c972449d1
Sep 27 12:46:33 dist-test-slave-lkn4 krb5kdc[5940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513193, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:46:33.484032 6024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:33.471281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38791 (local address 127.0.98.62:45241)
0927 12:46:33.471448 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:33.471452 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:33.472338 (+ 886us) server_negotiation.cc:407] Connection header received
0927 12:46:33.473803 (+ 1465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:33.473806 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:33.473859 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:33.473943 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:33.475383 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:33.476003 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:33.477601 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:33.477911 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:33.478016 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:33.480723 (+ 2707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:33.480726 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:33.480728 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:33.480755 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:33.482144 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:33.482669 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:33.482673 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:33.482674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:33.482756 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:33.483061 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:33.483064 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:33.483065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:33.483239 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:33.483332 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:33.483801 (+ 469us) server_negotiation.cc:299] Negotiation successful
0927 12:46:33.483906 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 12:46:33.484855 6446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:33.471674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45241 (local address 127.0.98.3:38791)
0927 12:46:33.472173 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:33.472219 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:33.473551 (+ 1332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:33.474225 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:33.474234 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:33.474704 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:33.475230 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:33.475245 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:33.476148 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:33.476152 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:33.477492 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:33.477503 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:33.478107 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:33.478113 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:33.478320 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:33.478958 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:33.478975 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:33.480526 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:33.482255 (+ 1729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:33.482261 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:33.482276 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:33.482575 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:33.482838 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:33.482841 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:33.482842 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:33.482993 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:33.483332 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:33.483337 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:33.483588 (+ 251us) client_negotiation.cc:769] Sending connection context
0927 12:46:33.483758 (+ 170us) client_negotiation.cc:240] Negotiation successful
0927 12:46:33.484029 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":115,"threads_started":1}
I20260927 12:46:33.485976 6444 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45241
I20260927 12:46:33.486254 6444 heartbeater.cc:464] Registering TS with master...
I20260927 12:46:33.486757 6444 heartbeater.cc:511] Master 127.0.98.62:45241 requested a full tablet report, sending...
I20260927 12:46:33.487762 5985 ts_manager.cc:194] Registered new tserver with Master: 6acdaeac2ecc4ebfae1e66311b7a3427 (127.0.98.3:43893)
I20260927 12:46:33.488297 5985 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38791
I20260927 12:46:33.490095 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/krb5kdc/test-admin.keytab.
Sep 27 12:46:33 dist-test-slave-lkn4 krb5kdc[5940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:46:33.507277 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260927 12:46:33.510001 6451 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:46:33.508663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55152 (local address 127.0.98.1:39903)
0927 12:46:33.508958 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:33.508971 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:33.508994 (+ 23us) server_negotiation.cc:407] Connection header received
0927 12:46:33.509045 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:33.509051 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:33.509263 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:33.509382 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:33.509848 (+ 466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55152: BlockingRecv error: recv got EOF from 127.0.0.1:55152 (error 108)
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":88,"threads_started":1}
W20260927 12:46:33.510128 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/krb5kdc/krb5cc' of type FILE
Sep 27 12:46:33 dist-test-slave-lkn4 krb5kdc[5940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:46:33.511173 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 12:46:33 dist-test-slave-lkn4 krb5kdc[5940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:46:33.512315 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 12:46:33 dist-test-slave-lkn4 krb5kdc[5940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 12:46:33.522498 6451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:33.513308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55158 (local address 127.0.98.1:39903)
0927 12:46:33.513453 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:33.513457 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:33.513471 (+ 14us) server_negotiation.cc:407] Connection header received
0927 12:46:33.513576 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:33.513578 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:33.513619 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:33.513687 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:33.514303 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:33.515121 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:33.515978 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:33.516423 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:33.516529 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:33.519030 (+ 2501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:33.519041 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:33.519057 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:33.519086 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:33.520676 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:33.521150 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:33.521156 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:33.521157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:33.521246 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:33.521491 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:33.521494 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:33.521496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:33.521759 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:33.521960 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:33.522185 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 12:46:33.522348 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260927 12:46:33.526571 6451 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:46:33.525491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55164 (local address 127.0.98.1:39903)
0927 12:46:33.525633 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:33.525638 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:33.525711 (+ 73us) server_negotiation.cc:407] Connection header received
0927 12:46:33.525858 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:33.525861 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:33.525904 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:33.525983 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:33.526467 (+ 484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55164: BlockingRecv error: recv got EOF from 127.0.0.1:55164 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260927 12:46:33.526979 392 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/krb5kdc/krb5cc)
Sep 27 12:46:33 dist-test-slave-lkn4 krb5kdc[5940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:46:33.527998 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 12:46:33 dist-test-slave-lkn4 krb5kdc[5940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:46:33.528853 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 12:46:33 dist-test-slave-lkn4 krb5kdc[5940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 12:46:33.537307 6451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:33.529734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55172 (local address 127.0.98.1:39903)
0927 12:46:33.529879 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:33.529884 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:33.529947 (+ 63us) server_negotiation.cc:407] Connection header received
0927 12:46:33.530095 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:33.530098 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:33.530158 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:33.530226 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:33.530719 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:33.531277 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:33.532059 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:33.532293 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:33.532382 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:33.534525 (+ 2143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:33.534528 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:33.534530 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:33.534552 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:33.535801 (+ 1249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:33.536204 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:33.536207 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:33.536208 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:33.536288 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:33.536522 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:33.536525 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:33.536527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:33.536695 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:33.536807 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:33.537039 (+ 232us) server_negotiation.cc:299] Negotiation successful
0927 12:46:33.537159 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260927 12:46:33.539881 6451 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:46:33.539089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55174 (local address 127.0.98.1:39903)
0927 12:46:33.539241 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:33.539244 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:33.539256 (+ 12us) server_negotiation.cc:407] Connection header received
0927 12:46:33.539353 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:33.539356 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:33.539387 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:33.539444 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:33.539797 (+ 353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55174: BlockingRecv error: recv got EOF from 127.0.0.1:55174 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260927 12:46:33.539997 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790513161160093-392-0/krb5kdc/krb5cc' of type FILE
Sep 27 12:46:33 dist-test-slave-lkn4 krb5kdc[5940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:46:33.540916 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 12:46:33 dist-test-slave-lkn4 krb5kdc[5940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:46:33.541697 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 12:46:33 dist-test-slave-lkn4 krb5kdc[5940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 12:46:33.549782 6451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:33.542479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55182 (local address 127.0.98.1:39903)
0927 12:46:33.542608 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:33.542611 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:33.542695 (+ 84us) server_negotiation.cc:407] Connection header received
0927 12:46:33.542754 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:33.542756 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:33.542792 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:33.542856 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:33.543255 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:33.543787 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:33.544501 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:33.544765 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:33.544850 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:33.547102 (+ 2252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:33.547105 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:33.547107 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:33.547129 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:33.548429 (+ 1300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:33.548783 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:33.548785 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:33.548786 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:33.548855 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:33.549095 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:33.549097 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:33.549098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:33.549236 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:33.549322 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:33.549523 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 12:46:33.549615 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 12:46:33.550683 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 6047
I20260927 12:46:33.555645 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 6181
I20260927 12:46:33.560217 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 6315
I20260927 12:46:33.564451 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 5953
2026-09-27T12:46:33Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2066 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:46:33 dist-test-slave-lkn4 krb5kdc[6464](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:46:33 dist-test-slave-lkn4 krb5kdc[6464](info): set up 2 sockets
Sep 27 12:46:33 dist-test-slave-lkn4 krb5kdc[6464](info): commencing operation
krb5kdc: starting...
W20260927 12:46:34.514302 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/krb5kdc/test-user.keytab.
Sep 27 12:46:34 dist-test-slave-lkn4 krb5kdc[6464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513194, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:46:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:46:34Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260927 12:46:34.633087 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40735
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40735
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:46:34.746923 6477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:46:34.747210 6477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:46:34.747265 6477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:46:34.751240 6477 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:46:34.751310 6477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:46:34.751339 6477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:46:34.751359 6477 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:46:34.751387 6477 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:46:34.756757 6477 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40735
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:46:34.757917 6477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:46:34.759063 6477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:46:34.767395 6483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:34.767464 6477 server_base.cc:1034] running on GCE node
W20260927 12:46:34.767659 6482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:34.767956 6485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:34.768537 6477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:46:34.769627 6477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:46:34.770778 6477 hybrid_clock.cc:648] HybridClock initialized: now 1790513194770758 us; error 33 us; skew 500 ppm
Sep 27 12:46:34 dist-test-slave-lkn4 krb5kdc[6464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513194, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:46:34.776681 6477 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260927 12:46:34.777627 6477 webserver.cc:460] Webserver started at http://127.0.98.62:35753/ using document root <none> and password file <none>
I20260927 12:46:34.778237 6477 fs_manager.cc:359] Metadata directory not provided
I20260927 12:46:34.778306 6477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:46:34.778507 6477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:46:34.780560 6477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/master-0/data/instance:
uuid: "205c8e3f55144799958e82a465ac9b66"
format_stamp: "Formatted at 2026-09-27 12:46:34 on dist-test-slave-lkn4"
server_key: "d27a9dec19ea47480de714c4780d5743"
server_key_iv: "7a208c757d727c799ad4765452084867"
server_key_version: "encryptionkey@0"
I20260927 12:46:34.781080 6477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/master-0/wal/instance:
uuid: "205c8e3f55144799958e82a465ac9b66"
format_stamp: "Formatted at 2026-09-27 12:46:34 on dist-test-slave-lkn4"
server_key: "d27a9dec19ea47480de714c4780d5743"
server_key_iv: "7a208c757d727c799ad4765452084867"
server_key_version: "encryptionkey@0"
I20260927 12:46:34.784739 6477 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 12:46:34.787333 6492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:34.788657 6477 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 12:46:34.788757 6477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/master-0/wal
uuid: "205c8e3f55144799958e82a465ac9b66"
format_stamp: "Formatted at 2026-09-27 12:46:34 on dist-test-slave-lkn4"
server_key: "d27a9dec19ea47480de714c4780d5743"
server_key_iv: "7a208c757d727c799ad4765452084867"
server_key_version: "encryptionkey@0"
I20260927 12:46:34.788901 6477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:46:34.803471 6477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:46:34.806784 6477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:46:34.806959 6477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:46:34.813691 6477 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40735
I20260927 12:46:34.813714 6544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40735 every 8 connection(s)
I20260927 12:46:34.814652 6477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:46:34.817564 6545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:46:34.819190 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 6477
I20260927 12:46:34.819308 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/master-0/wal/instance
I20260927 12:46:34.819558 392 external_mini_cluster.cc:1428] Setting key f850b7c633c06d6227cd3eee52277d69
I20260927 12:46:34.824059 6545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 205c8e3f55144799958e82a465ac9b66: Bootstrap starting.
I20260927 12:46:34.826736 6545 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 205c8e3f55144799958e82a465ac9b66: Neither blocks nor log segments found. Creating new log.
I20260927 12:46:34.827509 6545 log.cc:824] T 00000000000000000000000000000000 P 205c8e3f55144799958e82a465ac9b66: Log is configured to *not* fsync() on all Append() calls
Sep 27 12:46:34 dist-test-slave-lkn4 krb5kdc[6464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513194, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 12:46:34.829803 6545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 205c8e3f55144799958e82a465ac9b66: No bootstrap required, opened a new log
I20260927 12:46:34.833778 6545 raft_consensus.cc:348] T 00000000000000000000000000000000 P 205c8e3f55144799958e82a465ac9b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "205c8e3f55144799958e82a465ac9b66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40735 } }
I20260927 12:46:34.833961 6548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:34.820902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59084 (local address 127.0.98.62:40735)
0927 12:46:34.821314 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:34.821333 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:34.821375 (+ 42us) server_negotiation.cc:407] Connection header received
0927 12:46:34.822213 (+ 838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:34.822238 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:34.822653 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:34.823021 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:34.823628 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:34.825092 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:34.825974 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:34.826697 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:34.826859 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:34.829418 (+ 2559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:34.829424 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:34.829438 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:34.829464 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:34.831101 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:34.831588 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:34.831593 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:34.831597 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:34.831710 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:34.831987 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:34.831989 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:34.831990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:34.832423 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:34.832568 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:34.832868 (+ 300us) server_negotiation.cc:299] Negotiation successful
0927 12:46:34.833150 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":101,"threads_started":1}
I20260927 12:46:34.834009 6545 raft_consensus.cc:374] T 00000000000000000000000000000000 P 205c8e3f55144799958e82a465ac9b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:46:34.834080 6545 raft_consensus.cc:729] T 00000000000000000000000000000000 P 205c8e3f55144799958e82a465ac9b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 205c8e3f55144799958e82a465ac9b66, State: Initialized, Role: FOLLOWER
I20260927 12:46:34.834470 6545 consensus_queue.cc:260] T 00000000000000000000000000000000 P 205c8e3f55144799958e82a465ac9b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "205c8e3f55144799958e82a465ac9b66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40735 } }
I20260927 12:46:34.834596 6545 raft_consensus.cc:388] T 00000000000000000000000000000000 P 205c8e3f55144799958e82a465ac9b66 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:46:34.834669 6545 raft_consensus.cc:482] T 00000000000000000000000000000000 P 205c8e3f55144799958e82a465ac9b66 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:46:34.834767 6545 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 205c8e3f55144799958e82a465ac9b66 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:46:34.835740 6545 raft_consensus.cc:504] T 00000000000000000000000000000000 P 205c8e3f55144799958e82a465ac9b66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "205c8e3f55144799958e82a465ac9b66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40735 } }
I20260927 12:46:34.836093 6545 leader_election.cc:302] T 00000000000000000000000000000000 P 205c8e3f55144799958e82a465ac9b66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 205c8e3f55144799958e82a465ac9b66; no voters:
I20260927 12:46:34.836390 6545 leader_election.cc:288] T 00000000000000000000000000000000 P 205c8e3f55144799958e82a465ac9b66 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:46:34.836484 6550 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 205c8e3f55144799958e82a465ac9b66 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:46:34.836678 6550 raft_consensus.cc:686] T 00000000000000000000000000000000 P 205c8e3f55144799958e82a465ac9b66 [term 1 LEADER]: Becoming Leader. State: Replica: 205c8e3f55144799958e82a465ac9b66, State: Running, Role: LEADER
I20260927 12:46:34.837037 6550 consensus_queue.cc:237] T 00000000000000000000000000000000 P 205c8e3f55144799958e82a465ac9b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "205c8e3f55144799958e82a465ac9b66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40735 } }
I20260927 12:46:34.837505 6545 sys_catalog.cc:564] T 00000000000000000000000000000000 P 205c8e3f55144799958e82a465ac9b66 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:46:34.838500 6551 sys_catalog.cc:455] T 00000000000000000000000000000000 P 205c8e3f55144799958e82a465ac9b66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "205c8e3f55144799958e82a465ac9b66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "205c8e3f55144799958e82a465ac9b66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40735 } } }
I20260927 12:46:34.838622 6551 sys_catalog.cc:458] T 00000000000000000000000000000000 P 205c8e3f55144799958e82a465ac9b66 [sys.catalog]: This master's current role is: LEADER
I20260927 12:46:34.838990 6559 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:46:34.838948 6552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 205c8e3f55144799958e82a465ac9b66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 205c8e3f55144799958e82a465ac9b66. Latest consensus state: current_term: 1 leader_uuid: "205c8e3f55144799958e82a465ac9b66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "205c8e3f55144799958e82a465ac9b66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40735 } } }
I20260927 12:46:34.839085 6552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 205c8e3f55144799958e82a465ac9b66 [sys.catalog]: This master's current role is: LEADER
I20260927 12:46:34.841747 6559 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:46:34.847056 6559 catalog_manager.cc:1347] Generated new cluster ID: ee03576c83fa435283af71c4b0841e2f
I20260927 12:46:34.847111 6559 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:46:34.857932 6559 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:46:34.858984 6559 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:46:34.863633 6559 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 205c8e3f55144799958e82a465ac9b66: Generated new TSK 0
I20260927 12:46:34.864377 6559 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260927 12:46:34.896773 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40735
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:46:35.013010 6571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:46:35.013288 6571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:46:35.013330 6571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:46:35.017378 6571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:46:35.017452 6571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:46:35.017520 6571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:46:35.022931 6571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:46:35.024060 6571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:46:35.025136 6571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 12:46:35.035705 6571 server_base.cc:1034] running on GCE node
W20260927 12:46:35.035652 6576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:35.035624 6577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:35.035651 6579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:35.036309 6571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:46:35.036943 6571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:46:35.038116 6571 hybrid_clock.cc:648] HybridClock initialized: now 1790513195038108 us; error 36 us; skew 500 ppm
Sep 27 12:46:35 dist-test-slave-lkn4 krb5kdc[6464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513195, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:46:35.040966 6571 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 12:46:35.041868 6571 webserver.cc:460] Webserver started at http://127.0.98.1:42583/ using document root <none> and password file <none>
I20260927 12:46:35.042541 6571 fs_manager.cc:359] Metadata directory not provided
I20260927 12:46:35.042608 6571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:46:35.042778 6571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:46:35.044860 6571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-0/data/instance:
uuid: "9387b2fa57714be0aa7b8e69be342b8e"
format_stamp: "Formatted at 2026-09-27 12:46:35 on dist-test-slave-lkn4"
server_key: "80ca6b0de6bcbc890271b1c51712afab"
server_key_iv: "2a98600608c2265f5b6c0c30e12c1949"
server_key_version: "encryptionkey@0"
I20260927 12:46:35.045383 6571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9387b2fa57714be0aa7b8e69be342b8e"
format_stamp: "Formatted at 2026-09-27 12:46:35 on dist-test-slave-lkn4"
server_key: "80ca6b0de6bcbc890271b1c51712afab"
server_key_iv: "2a98600608c2265f5b6c0c30e12c1949"
server_key_version: "encryptionkey@0"
I20260927 12:46:35.049191 6571 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 12:46:35.051797 6586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:35.053166 6571 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 12:46:35.053277 6571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-0/wal
uuid: "9387b2fa57714be0aa7b8e69be342b8e"
format_stamp: "Formatted at 2026-09-27 12:46:35 on dist-test-slave-lkn4"
server_key: "80ca6b0de6bcbc890271b1c51712afab"
server_key_iv: "2a98600608c2265f5b6c0c30e12c1949"
server_key_version: "encryptionkey@0"
I20260927 12:46:35.053387 6571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:46:35.068271 6571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:46:35.071632 6571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:46:35.071820 6571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:46:35.072460 6571 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:46:35.073313 6571 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:46:35.073359 6571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:35.073401 6571 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:46:35.073429 6571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:35.082170 6571 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43975
I20260927 12:46:35.082218 6699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43975 every 8 connection(s)
I20260927 12:46:35.083117 6571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:46:35.083577 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 6571
I20260927 12:46:35.083706 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:46:35.084043 392 external_mini_cluster.cc:1428] Setting key aae04127cc9696a3285b9bef3d388581
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Sep 27 12:46:35 dist-test-slave-lkn4 krb5kdc[6464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513195, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 12:46:35.100171 6548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:35.084888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42889 (local address 127.0.98.62:40735)
0927 12:46:35.085025 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:35.085030 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:35.086085 (+ 1055us) server_negotiation.cc:407] Connection header received
0927 12:46:35.087811 (+ 1726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:35.087815 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:35.087874 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:35.087955 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:35.089178 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:35.089786 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:35.091006 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:35.091251 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:35.091345 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:35.096344 (+ 4999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:35.096349 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:35.096352 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:35.096385 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:35.098030 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:35.098586 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:35.098589 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:35.098591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:35.098682 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:35.098977 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:35.098980 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:35.098982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:35.099231 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:35.099328 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:35.099849 (+ 521us) server_negotiation.cc:299] Negotiation successful
0927 12:46:35.099994 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 12:46:35.100906 6702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:35.085388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40735 (local address 127.0.98.1:42889)
0927 12:46:35.085894 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:35.085960 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:35.087588 (+ 1628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:35.088138 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:35.088145 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:35.088523 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:35.089014 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:35.089028 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:35.089916 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:35.089919 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:35.090879 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:35.090889 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:35.091437 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:35.091443 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:35.091607 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:35.092261 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:35.092284 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:35.096141 (+ 3857us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:35.098161 (+ 2020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:35.098167 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:35.098181 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:35.098474 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:35.098759 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:35.098761 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:35.098763 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:35.098893 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:35.099339 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:35.099345 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:35.099597 (+ 252us) client_negotiation.cc:769] Sending connection context
0927 12:46:35.099751 (+ 154us) client_negotiation.cc:240] Negotiation successful
0927 12:46:35.100032 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":112,"threads_started":1}
I20260927 12:46:35.102128 6700 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40735
I20260927 12:46:35.102372 6700 heartbeater.cc:464] Registering TS with master...
I20260927 12:46:35.102859 6700 heartbeater.cc:511] Master 127.0.98.62:40735 requested a full tablet report, sending...
I20260927 12:46:35.104414 6509 ts_manager.cc:194] Registered new tserver with Master: 9387b2fa57714be0aa7b8e69be342b8e (127.0.98.1:43975)
I20260927 12:46:35.105610 6509 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42889
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260927 12:46:35.111310 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40735
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:46:35.223081 6705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:46:35.223341 6705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:46:35.223384 6705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:46:35.227443 6705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:46:35.227506 6705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:46:35.227576 6705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:46:35.233001 6705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:46:35.234033 6705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:46:35.235087 6705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:46:35.244305 6711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:35.244349 6713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:35.244299 6710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:35.244585 6705 server_base.cc:1034] running on GCE node
I20260927 12:46:35.245048 6705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:46:35.245664 6705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:46:35.246829 6705 hybrid_clock.cc:648] HybridClock initialized: now 1790513195246799 us; error 43 us; skew 500 ppm
Sep 27 12:46:35 dist-test-slave-lkn4 krb5kdc[6464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513195, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:46:35.249451 6705 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 12:46:35.250334 6705 webserver.cc:460] Webserver started at http://127.0.98.2:40097/ using document root <none> and password file <none>
I20260927 12:46:35.250917 6705 fs_manager.cc:359] Metadata directory not provided
I20260927 12:46:35.250981 6705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:46:35.251158 6705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:46:35.253177 6705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-1/data/instance:
uuid: "5752b93fbf53475cb5cf0593fd908fed"
format_stamp: "Formatted at 2026-09-27 12:46:35 on dist-test-slave-lkn4"
server_key: "d7b9e5dabc505f2e79af83161095d7e6"
server_key_iv: "b899991ee2f1f1ed3560523b7515d7bd"
server_key_version: "encryptionkey@0"
I20260927 12:46:35.253707 6705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5752b93fbf53475cb5cf0593fd908fed"
format_stamp: "Formatted at 2026-09-27 12:46:35 on dist-test-slave-lkn4"
server_key: "d7b9e5dabc505f2e79af83161095d7e6"
server_key_iv: "b899991ee2f1f1ed3560523b7515d7bd"
server_key_version: "encryptionkey@0"
I20260927 12:46:35.257364 6705 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 12:46:35.259763 6720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:35.261062 6705 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 12:46:35.261230 6705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-1/wal
uuid: "5752b93fbf53475cb5cf0593fd908fed"
format_stamp: "Formatted at 2026-09-27 12:46:35 on dist-test-slave-lkn4"
server_key: "d7b9e5dabc505f2e79af83161095d7e6"
server_key_iv: "b899991ee2f1f1ed3560523b7515d7bd"
server_key_version: "encryptionkey@0"
I20260927 12:46:35.261336 6705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:46:35.275977 6705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:46:35.279105 6705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:46:35.279276 6705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:46:35.279862 6705 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:46:35.280671 6705 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:46:35.280740 6705 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:35.280820 6705 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:46:35.280848 6705 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:35.289590 6705 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41427
I20260927 12:46:35.289615 6833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41427 every 8 connection(s)
I20260927 12:46:35.290547 6705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:46:35.297849 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 6705
I20260927 12:46:35.297972 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:46:35.298305 392 external_mini_cluster.cc:1428] Setting key fd93cff0967a75045385a93c3abffdcc
Sep 27 12:46:35 dist-test-slave-lkn4 krb5kdc[6464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513195, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 12:46:35.305157 6548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:35.292353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39249 (local address 127.0.98.62:40735)
0927 12:46:35.292521 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:35.292526 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:35.293469 (+ 943us) server_negotiation.cc:407] Connection header received
0927 12:46:35.294884 (+ 1415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:35.294887 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:35.294937 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:35.295011 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:35.296210 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:35.296815 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:35.298161 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:35.298398 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:35.298492 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:35.301201 (+ 2709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:35.301205 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:35.301208 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:35.301237 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:35.302804 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:35.303451 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:35.303454 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:35.303455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:35.303529 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:35.303943 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:35.303946 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:35.303948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:35.304159 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:35.304242 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:35.304896 (+ 654us) server_negotiation.cc:299] Negotiation successful
0927 12:46:35.305001 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 12:46:35.306501 6836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:35.292828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40735 (local address 127.0.98.2:39249)
0927 12:46:35.293328 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:35.293373 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:35.294638 (+ 1265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:35.295185 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:35.295192 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:35.295565 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:35.296061 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:35.296075 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:35.296937 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:35.296940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:35.298028 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:35.298065 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:35.298551 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:35.298556 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:35.298778 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:35.299384 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:35.299405 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:35.301016 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:35.302927 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:35.302934 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:35.302951 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:35.303348 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:35.303614 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:35.303617 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:35.303619 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:35.303818 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:35.304270 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:35.304277 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:35.304622 (+ 345us) client_negotiation.cc:769] Sending connection context
0927 12:46:35.304863 (+ 241us) client_negotiation.cc:240] Negotiation successful
0927 12:46:35.305286 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":110,"threads_started":1}
I20260927 12:46:35.307981 6834 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40735
I20260927 12:46:35.308248 6834 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 12:46:35.308761 6834 heartbeater.cc:511] Master 127.0.98.62:40735 requested a full tablet report, sending...
I20260927 12:46:35.309824 6509 ts_manager.cc:194] Registered new tserver with Master: 5752b93fbf53475cb5cf0593fd908fed (127.0.98.2:41427)
I20260927 12:46:35.310420 6509 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39249
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 12:46:35.324999 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40735
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:46:35.432183 6839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:46:35.432492 6839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:46:35.432538 6839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:46:35.436740 6839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:46:35.436822 6839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:46:35.436882 6839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:46:35.442555 6839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:46:35.443634 6839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:46:35.444675 6839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:46:35.454344 6844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:35.454375 6845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:35.454335 6847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:35.454661 6839 server_base.cc:1034] running on GCE node
I20260927 12:46:35.455145 6839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:46:35.455806 6839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:46:35.457000 6839 hybrid_clock.cc:648] HybridClock initialized: now 1790513195456977 us; error 39 us; skew 500 ppm
Sep 27 12:46:35 dist-test-slave-lkn4 krb5kdc[6464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513195, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:46:35.459894 6839 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 12:46:35.460878 6839 webserver.cc:460] Webserver started at http://127.0.98.3:40971/ using document root <none> and password file <none>
I20260927 12:46:35.461432 6839 fs_manager.cc:359] Metadata directory not provided
I20260927 12:46:35.461498 6839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:46:35.461705 6839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:46:35.463925 6839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-2/data/instance:
uuid: "ae19c5cb5e6d4abc8d5bf1463c817642"
format_stamp: "Formatted at 2026-09-27 12:46:35 on dist-test-slave-lkn4"
server_key: "d73887f83eb19c4cda2060ed11a28650"
server_key_iv: "97f4c863c38b37e71eab4934e33b58a3"
server_key_version: "encryptionkey@0"
I20260927 12:46:35.464553 6839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ae19c5cb5e6d4abc8d5bf1463c817642"
format_stamp: "Formatted at 2026-09-27 12:46:35 on dist-test-slave-lkn4"
server_key: "d73887f83eb19c4cda2060ed11a28650"
server_key_iv: "97f4c863c38b37e71eab4934e33b58a3"
server_key_version: "encryptionkey@0"
I20260927 12:46:35.468504 6839 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260927 12:46:35.471280 6854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:35.472765 6839 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 12:46:35.472899 6839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-2/wal
uuid: "ae19c5cb5e6d4abc8d5bf1463c817642"
format_stamp: "Formatted at 2026-09-27 12:46:35 on dist-test-slave-lkn4"
server_key: "d73887f83eb19c4cda2060ed11a28650"
server_key_iv: "97f4c863c38b37e71eab4934e33b58a3"
server_key_version: "encryptionkey@0"
I20260927 12:46:35.473016 6839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:46:35.487753 6839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:46:35.491282 6839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:46:35.491494 6839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:46:35.492139 6839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:46:35.493064 6839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:46:35.493119 6839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:35.493178 6839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:46:35.493212 6839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:35.502662 6839 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45825
I20260927 12:46:35.502677 6967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45825 every 8 connection(s)
I20260927 12:46:35.503672 6839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:46:35.511098 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 6839
I20260927 12:46:35.511221 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:46:35.511466 392 external_mini_cluster.cc:1428] Setting key fd12add2149bb666f00a4ac73b88ac7a
Sep 27 12:46:35 dist-test-slave-lkn4 krb5kdc[6464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513195, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 12:46:35.517504 6548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:35.505580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53859 (local address 127.0.98.62:40735)
0927 12:46:35.505861 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:35.505866 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:35.506707 (+ 841us) server_negotiation.cc:407] Connection header received
0927 12:46:35.508087 (+ 1380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:35.508090 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:35.508138 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:35.508214 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:35.509423 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:35.509952 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:35.511190 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:35.511441 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:35.511531 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:35.514164 (+ 2633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:35.514167 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:35.514169 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:35.514194 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:35.515541 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:35.516121 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:35.516124 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:35.516126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:35.516205 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:35.516507 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:35.516509 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:35.516510 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:35.516683 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:35.516751 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:35.517280 (+ 529us) server_negotiation.cc:299] Negotiation successful
0927 12:46:35.517384 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176}
I20260927 12:46:35.518433 6970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:35.506067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40735 (local address 127.0.98.3:53859)
0927 12:46:35.506552 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:35.506595 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:35.507867 (+ 1272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:35.508406 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:35.508413 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:35.508781 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:35.509281 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:35.509296 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:35.510064 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:35.510067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:35.511066 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:35.511077 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:35.511594 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:35.511598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:35.511791 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:35.512397 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:35.512414 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:35.513991 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:35.515671 (+ 1680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:35.515677 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:35.515730 (+ 53us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:35.516010 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:35.516285 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:35.516288 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:35.516289 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:35.516436 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:35.516795 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:35.516801 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:35.517067 (+ 266us) client_negotiation.cc:769] Sending connection context
0927 12:46:35.517223 (+ 156us) client_negotiation.cc:240] Negotiation successful
0927 12:46:35.517496 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":145,"threads_started":1}
I20260927 12:46:35.519702 6968 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40735
I20260927 12:46:35.519989 6968 heartbeater.cc:464] Registering TS with master...
I20260927 12:46:35.520550 6968 heartbeater.cc:511] Master 127.0.98.62:40735 requested a full tablet report, sending...
I20260927 12:46:35.521639 6509 ts_manager.cc:194] Registered new tserver with Master: ae19c5cb5e6d4abc8d5bf1463c817642 (127.0.98.3:45825)
I20260927 12:46:35.522253 6509 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53859
I20260927 12:46:35.525134 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260927 12:46:35.552953 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:41239
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40735,127.0.98.61:41239
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:46:35.662356 6973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:46:35.662667 6973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:46:35.662726 6973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:46:35.667153 6973 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:46:35.667230 6973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:46:35.667253 6973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:46:35.667268 6973 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:46:35.667285 6973 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:46:35.672847 6973 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40735,127.0.98.61:41239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:41239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
W20260927 12:46:35.673069 6973 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:40735,127.0.98.61:41239'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260927 12:46:35.673980 6973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:46:35.675005 6973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:46:35.683413 6979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:35.683398 6978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:35.683498 6973 server_base.cc:1034] running on GCE node
W20260927 12:46:35.683416 6981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:35.684327 6973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:46:35.685496 6973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:46:35.686748 6973 hybrid_clock.cc:648] HybridClock initialized: now 1790513195686726 us; error 38 us; skew 500 ppm
Sep 27 12:46:35 dist-test-slave-lkn4 krb5kdc[6464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513195, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:46:35.689556 6973 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260927 12:46:35.690543 6973 webserver.cc:460] Webserver started at http://127.0.98.61:41101/ using document root <none> and password file <none>
I20260927 12:46:35.691097 6973 fs_manager.cc:359] Metadata directory not provided
I20260927 12:46:35.691164 6973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:46:35.691370 6973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:46:35.693548 6973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/master-1/data/instance:
uuid: "4fdbce7f7c3d4fda8c562a4fb4072671"
format_stamp: "Formatted at 2026-09-27 12:46:35 on dist-test-slave-lkn4"
server_key: "9d2a09369bc4eb124f11f8091f5f722d"
server_key_iv: "5a798cef9aa445558128f942026d8607"
server_key_version: "encryptionkey@0"
I20260927 12:46:35.694137 6973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/master-1/wal/instance:
uuid: "4fdbce7f7c3d4fda8c562a4fb4072671"
format_stamp: "Formatted at 2026-09-27 12:46:35 on dist-test-slave-lkn4"
server_key: "9d2a09369bc4eb124f11f8091f5f722d"
server_key_iv: "5a798cef9aa445558128f942026d8607"
server_key_version: "encryptionkey@0"
I20260927 12:46:35.697940 6973 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 12:46:35.700635 6988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:35.702005 6973 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 12:46:35.702174 6973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/master-1/wal
uuid: "4fdbce7f7c3d4fda8c562a4fb4072671"
format_stamp: "Formatted at 2026-09-27 12:46:35 on dist-test-slave-lkn4"
server_key: "9d2a09369bc4eb124f11f8091f5f722d"
server_key_iv: "5a798cef9aa445558128f942026d8607"
server_key_version: "encryptionkey@0"
I20260927 12:46:35.702288 6973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:46:35.732371 6973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:46:35.735524 6973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:46:35.735682 6973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:46:35.742129 6973 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:41239
I20260927 12:46:35.742377 7040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:41239 every 8 connection(s)
I20260927 12:46:35.743085 6973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/master-1/data/info.pb
I20260927 12:46:35.745754 7041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:46:35.749450 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 6973
I20260927 12:46:35.749627 7041 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40735 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 12:46:35 dist-test-slave-lkn4 krb5kdc[6464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513195, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 27 12:46:35 dist-test-slave-lkn4 krb5kdc[6464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513194, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 12:46:35.762542 6548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:35.751755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59098 (local address 127.0.98.62:40735)
0927 12:46:35.751879 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:35.751883 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:35.752132 (+ 249us) server_negotiation.cc:407] Connection header received
0927 12:46:35.752559 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:35.752562 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:35.752609 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:35.752685 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:35.753582 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:35.754229 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:35.755311 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:35.755555 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:35.755634 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:35.758608 (+ 2974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:35.758611 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:35.758614 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:35.758640 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:35.760206 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:35.760890 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:35.760893 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:35.760894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:35.760967 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:35.761368 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:35.761371 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:35.761373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:35.761639 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:35.761734 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:35.762254 (+ 520us) server_negotiation.cc:299] Negotiation successful
0927 12:46:35.762377 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 12:46:35.763509 7044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:35.750978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48028 (local address 127.0.98.61:41239)
0927 12:46:35.751381 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:35.751395 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:35.751428 (+ 33us) server_negotiation.cc:407] Connection header received
0927 12:46:35.752305 (+ 877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:35.752327 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:35.752581 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:35.752747 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:35.753504 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:35.754522 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:35.755325 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:35.755746 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:35.755845 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:35.759037 (+ 3192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:35.759045 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:35.759059 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:35.759083 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:35.760735 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:35.761219 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:35.761224 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:35.761225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:35.761290 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:35.761553 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:35.761556 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:35.761558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:35.761978 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:35.762159 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:35.762415 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 12:46:35.762595 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":84,"server-negotiator.queue_time_us":266,"thread_start_us":115,"threads_started":1}
I20260927 12:46:35.763512 7045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:35.751641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40735 (local address 127.0.0.1:59098)
0927 12:46:35.751987 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:35.752040 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:35.752371 (+ 331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:35.752743 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:35.752748 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:35.753181 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:35.753438 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:35.753455 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:35.754345 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:35.754348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:35.755193 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:35.755201 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:35.755691 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:35.755694 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:35.755836 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:35.756614 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:35.756640 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:35.758449 (+ 1809us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:35.760348 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:35.760356 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:35.760365 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:35.760770 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:35.761052 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:35.761055 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:35.761057 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:35.761257 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:35.761753 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:35.761759 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:35.761986 (+ 227us) client_negotiation.cc:769] Sending connection context
0927 12:46:35.762195 (+ 209us) client_negotiation.cc:240] Negotiation successful
0927 12:46:35.762541 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":6,"thread_start_us":78,"threads_started":1}
I20260927 12:46:35.765426 7041 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41239 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 12:46:35 dist-test-slave-lkn4 krb5kdc[6464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513195, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 12:46:35.773056 7045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:35.765934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41239 (local address 127.0.0.1:48044)
0927 12:46:35.766106 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:35.766117 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:35.766181 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:35.766428 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:35.766429 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:35.766578 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:35.766671 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:35.766676 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:35.767396 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:35.767399 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:35.767971 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:35.767979 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:35.768414 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:35.768417 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:35.768494 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:35.769000 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:35.769011 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:35.770339 (+ 1328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:35.771798 (+ 1459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:35.771801 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:35.771803 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:35.771983 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:35.772268 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:35.772270 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:35.772271 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:35.772338 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:35.772649 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:35.772650 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:35.772720 (+ 70us) client_negotiation.cc:769] Sending connection context
0927 12:46:35.772793 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 12:46:35.772898 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260927 12:46:35.773072 7044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:35.765992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48044 (local address 127.0.98.61:41239)
0927 12:46:35.766121 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:35.766125 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:35.766150 (+ 25us) server_negotiation.cc:407] Connection header received
0927 12:46:35.766283 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:35.766286 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:35.766324 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:35.766400 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:35.766821 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:35.767307 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:35.768058 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:35.768306 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:35.768380 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:35.770461 (+ 2081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:35.770464 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:35.770466 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:35.770486 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:35.771699 (+ 1213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:35.772090 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:35.772093 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:35.772094 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:35.772175 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:35.772422 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:35.772424 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:35.772425 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:35.772587 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:35.772659 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:35.772836 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 12:46:35.772951 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":37}
I20260927 12:46:35.776145 7041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fdbce7f7c3d4fda8c562a4fb4072671: Bootstrap starting.
I20260927 12:46:35.778533 7041 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4fdbce7f7c3d4fda8c562a4fb4072671: Neither blocks nor log segments found. Creating new log.
I20260927 12:46:35.779268 7041 log.cc:824] T 00000000000000000000000000000000 P 4fdbce7f7c3d4fda8c562a4fb4072671: Log is configured to *not* fsync() on all Append() calls
I20260927 12:46:35.781255 7041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fdbce7f7c3d4fda8c562a4fb4072671: No bootstrap required, opened a new log
I20260927 12:46:35.784632 7041 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4fdbce7f7c3d4fda8c562a4fb4072671 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205c8e3f55144799958e82a465ac9b66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40735 } } peers { permanent_uuid: "4fdbce7f7c3d4fda8c562a4fb4072671" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41239 } }
I20260927 12:46:35.784901 7041 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4fdbce7f7c3d4fda8c562a4fb4072671 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:46:35.784969 7041 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4fdbce7f7c3d4fda8c562a4fb4072671 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fdbce7f7c3d4fda8c562a4fb4072671, State: Initialized, Role: FOLLOWER
I20260927 12:46:35.785430 7041 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fdbce7f7c3d4fda8c562a4fb4072671 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205c8e3f55144799958e82a465ac9b66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40735 } } peers { permanent_uuid: "4fdbce7f7c3d4fda8c562a4fb4072671" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41239 } }
I20260927 12:46:35.785969 7047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fdbce7f7c3d4fda8c562a4fb4072671 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205c8e3f55144799958e82a465ac9b66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40735 } } peers { permanent_uuid: "4fdbce7f7c3d4fda8c562a4fb4072671" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41239 } } }
I20260927 12:46:35.786151 7047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fdbce7f7c3d4fda8c562a4fb4072671 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:46:35.786857 7041 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fdbce7f7c3d4fda8c562a4fb4072671 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 12:46:35.791388 7058 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4fdbce7f7c3d4fda8c562a4fb4072671: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 12:46:35.791457 7058 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 12:46:35.793685 6973 master_runner.cc:428] Detected that this master 4fdbce7f7c3d4fda8c562a4fb4072671 is joining an existing cluster
I20260927 12:46:35.793741 6973 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:41239
I20260927 12:46:35.804150 7044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:35.796128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48048 (local address 127.0.98.61:41239)
0927 12:46:35.796294 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:35.796299 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:35.796313 (+ 14us) server_negotiation.cc:407] Connection header received
0927 12:46:35.796418 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:35.796422 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:35.796467 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:35.796549 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:35.798217 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:35.798927 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:35.799862 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:35.800181 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:35.800290 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:35.801353 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:35.801356 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:35.801359 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:35.801382 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:35.802813 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:35.803132 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:35.803135 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:35.803137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:35.803200 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:35.803414 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:35.803417 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:35.803418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:35.803567 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:35.803653 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:35.803860 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 12:46:35.803996 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":95}
I20260927 12:46:35.804333 6548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:35.795780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59100 (local address 127.0.98.62:40735)
0927 12:46:35.795897 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:35.795901 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:35.796190 (+ 289us) server_negotiation.cc:407] Connection header received
0927 12:46:35.796348 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:35.796351 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:35.796400 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:35.796447 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:35.797241 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:35.798064 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:35.798916 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:35.799230 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:35.799331 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:35.800574 (+ 1243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:35.800579 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:35.800582 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:35.800616 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:35.802326 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:35.802856 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:35.802860 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:35.802863 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:35.802972 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:35.803308 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:35.803312 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:35.803314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:35.803536 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:35.803628 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:35.804031 (+ 403us) server_negotiation.cc:299] Negotiation successful
0927 12:46:35.804160 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 12:46:35.804158 7065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:35.795787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41239 (local address 127.0.0.1:48048)
0927 12:46:35.796255 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:35.796267 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:35.796335 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:35.796575 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:35.796577 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:35.796883 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:35.797118 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:35.797123 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:35.799025 (+ 1902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:35.799028 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:35.799680 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:35.799691 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:35.800301 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:35.800305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:35.800395 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:35.800838 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:35.800849 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:35.801244 (+ 395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:35.802891 (+ 1647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:35.802894 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:35.802896 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:35.803058 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:35.803272 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:35.803275 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:35.803277 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:35.803346 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:35.803650 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:35.803654 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:35.803731 (+ 77us) client_negotiation.cc:769] Sending connection context
0927 12:46:35.803812 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 12:46:35.803950 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"mutex_wait_us":72,"spinlock_wait_cycles":7680,"thread_start_us":120,"threads_started":1}
I20260927 12:46:35.805047 7064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:35.795713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40735 (local address 127.0.0.1:59100)
0927 12:46:35.796092 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:35.796106 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:35.796224 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:35.796575 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:35.796577 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:35.796880 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:35.797115 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:35.797123 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:35.798177 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:35.798180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:35.798809 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:35.798817 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:35.799362 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:35.799367 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:35.799465 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:35.800027 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:35.800038 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:35.800427 (+ 389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:35.802452 (+ 2025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:35.802455 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:35.802457 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:35.802759 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:35.803096 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:35.803100 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:35.803102 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:35.803200 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:35.803705 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:35.803708 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:35.803812 (+ 104us) client_negotiation.cc:769] Sending connection context
0927 12:46:35.804538 (+ 726us) client_negotiation.cc:240] Negotiation successful
0927 12:46:35.804842 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":40,"spinlock_wait_cycles":768,"thread_start_us":100,"threads_started":1}
Sep 27 12:46:35 dist-test-slave-lkn4 krb5kdc[6464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513194, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 12:46:35.820112 7044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:35.809782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48056 (local address 127.0.98.61:41239)
0927 12:46:35.809901 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:35.809906 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:35.810265 (+ 359us) server_negotiation.cc:407] Connection header received
0927 12:46:35.810463 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:35.810466 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:35.810524 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:35.810625 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:35.811331 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:35.812090 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:35.813027 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:35.813365 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:35.813574 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:35.816539 (+ 2965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:35.816543 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:35.816545 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:35.816571 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:35.818227 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:35.818802 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:35.818806 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:35.818808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:35.818894 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:35.819174 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:35.819177 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:35.819179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:35.819394 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:35.819478 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:35.819841 (+ 363us) server_negotiation.cc:299] Negotiation successful
0927 12:46:35.819958 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 12:46:35.820173 7066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:35.809745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41239 (local address 127.0.0.1:48056)
0927 12:46:35.810140 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:35.810193 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:35.810328 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:35.810648 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:35.810653 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:35.811088 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:35.811196 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:35.811202 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:35.812219 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:35.812222 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:35.812902 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:35.812919 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:35.813498 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:35.813502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:35.813618 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:35.814487 (+ 869us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:35.814501 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:35.816370 (+ 1869us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:35.818363 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:35.818373 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:35.818379 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:35.818674 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:35.818984 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:35.818986 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:35.818988 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:35.819093 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:35.819489 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:35.819495 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:35.819630 (+ 135us) client_negotiation.cc:769] Sending connection context
0927 12:46:35.819789 (+ 159us) client_negotiation.cc:240] Negotiation successful
0927 12:46:35.819998 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":109,"threads_started":1}
I20260927 12:46:35.821822 6509 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:41239: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "4fdbce7f7c3d4fda8c562a4fb4072671"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 41239
}
attrs {
promote: true
}
}
dest_uuid: "205c8e3f55144799958e82a465ac9b66"
cas_config_opid_index: -1
I20260927 12:46:35.822538 6509 consensus_queue.cc:237] T 00000000000000000000000000000000 P 205c8e3f55144799958e82a465ac9b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "205c8e3f55144799958e82a465ac9b66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40735 } } peers { permanent_uuid: "4fdbce7f7c3d4fda8c562a4fb4072671" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41239 } attrs { promote: true } }
I20260927 12:46:35.824402 7068 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 205c8e3f55144799958e82a465ac9b66 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 4fdbce7f7c3d4fda8c562a4fb4072671 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "205c8e3f55144799958e82a465ac9b66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40735 } } peers { permanent_uuid: "4fdbce7f7c3d4fda8c562a4fb4072671" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41239 } attrs { promote: true } } }
I20260927 12:46:35.825062 7015 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fdbce7f7c3d4fda8c562a4fb4072671 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:46:35.826326 7068 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:41239
I20260927 12:46:35.826406 7015 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4fdbce7f7c3d4fda8c562a4fb4072671 [term 1 FOLLOWER]: Refusing update from remote peer 205c8e3f55144799958e82a465ac9b66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260927 12:46:35.826787 6973 master.cc:537] Master@127.0.98.61:41239 shutting down...
I20260927 12:46:35.828424 7068 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 205c8e3f55144799958e82a465ac9b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fdbce7f7c3d4fda8c562a4fb4072671" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41239 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 12:46:35.830621 6493 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260927 12:46:35.830780 6973 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 4fdbce7f7c3d4fda8c562a4fb4072671 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:46:35.828773 7070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 205c8e3f55144799958e82a465ac9b66 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "205c8e3f55144799958e82a465ac9b66" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "205c8e3f55144799958e82a465ac9b66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40735 } } peers { permanent_uuid: "4fdbce7f7c3d4fda8c562a4fb4072671" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41239 } attrs { promote: true } } }
I20260927 12:46:35.831046 6973 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 4fdbce7f7c3d4fda8c562a4fb4072671 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:46:35.831102 7070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 205c8e3f55144799958e82a465ac9b66 [sys.catalog]: This master's current role is: LEADER
I20260927 12:46:35.831146 6973 tablet_replica.cc:330] T 00000000000000000000000000000000 P 4fdbce7f7c3d4fda8c562a4fb4072671: stopping tablet replica
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260927 12:46:35.836437 7066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:35.836148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41239 (local address 127.0.0.1:48064)
0927 12:46:35.836284 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:35.836351 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 12:46:35.836582 6493 consensus_peers.cc:487] T 00000000000000000000000000000000 P 205c8e3f55144799958e82a465ac9b66 -> Peer 4fdbce7f7c3d4fda8c562a4fb4072671 (127.0.98.61:41239): Couldn't send request to peer 4fdbce7f7c3d4fda8c562a4fb4072671. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41239: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 12:46:35.837263 6973 master.cc:552] Master@127.0.98.61:41239 shutdown complete.
I20260927 12:46:35.837311 6973 master_runner.cc:305] Clearing existing system tablet
I20260927 12:46:35.837543 6973 fs_manager.cc:359] Metadata directory not provided
I20260927 12:46:35.837598 6973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
I20260927 12:46:35.838928 6973 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 12:46:35.846004 7075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:35.846333 6973 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:46:35.846489 6973 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.000s
I20260927 12:46:35.846534 6973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/master-1/wal
uuid: "4fdbce7f7c3d4fda8c562a4fb4072671"
format_stamp: "Formatted at 2026-09-27 12:46:35 on dist-test-slave-lkn4"
server_key: "9d2a09369bc4eb124f11f8091f5f722d"
server_key_iv: "5a798cef9aa445558128f942026d8607"
server_key_version: "encryptionkey@0"
I20260927 12:46:35.848412 6973 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 4fdbce7f7c3d4fda8c562a4fb4072671: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:46:35.850612 6973 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 4fdbce7f7c3d4fda8c562a4fb4072671: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 12:46:35.850692 6973 log.cc:1196] T 00000000000000000000000000000000 P 4fdbce7f7c3d4fda8c562a4fb4072671: Deleting WAL directory at /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260927 12:46:35.850921 6973 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 4fdbce7f7c3d4fda8c562a4fb4072671: Deleting consensus metadata
I20260927 12:46:35.851076 6973 master_runner.cc:315] Copying system tablet from 127.0.98.62:40735
I20260927 12:46:35.852108 6973 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 4fdbce7f7c3d4fda8c562a4fb4072671: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:40735
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260927 12:46:35.853596 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45213
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40735,127.0.98.61:41239,127.0.98.60:45213
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 12:46:35.859830 6548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:35.852699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59110 (local address 127.0.98.62:40735)
0927 12:46:35.853117 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:35.853121 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:35.853139 (+ 18us) server_negotiation.cc:407] Connection header received
0927 12:46:35.853184 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:35.853188 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:35.853235 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:35.853329 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:35.853847 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:35.854591 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:35.855517 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:35.855748 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:35.855833 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:35.856895 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:35.856898 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:35.856901 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:35.856926 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:35.858377 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:35.858747 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:35.858751 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:35.858753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:35.858825 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:35.859057 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:35.859060 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:35.859062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:35.859251 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:35.859333 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:35.859565 (+ 232us) server_negotiation.cc:299] Negotiation successful
0927 12:46:35.859674 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326}
I20260927 12:46:35.859839 7081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:35.852627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40735 (local address 127.0.0.1:59110)
0927 12:46:35.852866 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:35.852880 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:35.852974 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:35.853368 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:35.853372 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:35.853568 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:35.853702 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:35.853710 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:35.854731 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:35.854734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:35.855409 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:35.855417 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:35.855847 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:35.855850 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:35.855938 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:35.856403 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:35.856417 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:35.856778 (+ 361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:35.858474 (+ 1696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:35.858477 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:35.858479 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:35.858664 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:35.858902 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:35.858905 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:35.858907 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:35.858983 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:35.859335 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:35.859339 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:35.859427 (+ 88us) client_negotiation.cc:769] Sending connection context
0927 12:46:35.859509 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 12:46:35.859654 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":85,"threads_started":1}
I20260927 12:46:35.860670 6529 tablet_copy_service.cc:140] P 205c8e3f55144799958e82a465ac9b66: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 4fdbce7f7c3d4fda8c562a4fb4072671 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:59110)
I20260927 12:46:35.860827 6529 tablet_copy_service.cc:161] P 205c8e3f55144799958e82a465ac9b66: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 4fdbce7f7c3d4fda8c562a4fb4072671 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:59110: session id = 4fdbce7f7c3d4fda8c562a4fb4072671-00000000000000000000000000000000
I20260927 12:46:35.862269 6529 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 205c8e3f55144799958e82a465ac9b66: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 12:46:35.863734 6973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 12:46:35.865584 6973 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 4fdbce7f7c3d4fda8c562a4fb4072671: tablet copy: Starting download of 0 data blocks...
I20260927 12:46:35.865757 6973 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 4fdbce7f7c3d4fda8c562a4fb4072671: tablet copy: Starting download of 1 WAL segments...
I20260927 12:46:35.867262 6973 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 4fdbce7f7c3d4fda8c562a4fb4072671: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 12:46:35.868724 6529 tablet_copy_service.cc:342] P 205c8e3f55144799958e82a465ac9b66: Request end of tablet copy session 4fdbce7f7c3d4fda8c562a4fb4072671-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:59110
I20260927 12:46:35.868822 6529 tablet_copy_service.cc:433] P 205c8e3f55144799958e82a465ac9b66: ending tablet copy session 4fdbce7f7c3d4fda8c562a4fb4072671-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 4fdbce7f7c3d4fda8c562a4fb4072671
I20260927 12:46:35.875444 6973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 12:46:35.877205 6973 server_base.cc:1034] running on GCE node
W20260927 12:46:35.877192 7086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:35.877180 7083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:35.877224 7084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:35.877696 6973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:46:35.878010 6973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:46:35.879217 6973 hybrid_clock.cc:648] HybridClock initialized: now 1790513195879204 us; error 34 us; skew 500 ppm
Sep 27 12:46:35 dist-test-slave-lkn4 krb5kdc[6464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513195, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:46:35.880558 6973 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260927 12:46:35.880980 6973 webserver.cc:460] Webserver started at http://127.0.98.61:39147/ using document root <none> and password file <none>
I20260927 12:46:35.881181 6973 fs_manager.cc:359] Metadata directory not provided
I20260927 12:46:35.881219 6973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:46:35.882296 6973 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 12:46:35.883132 7093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:35.883334 6973 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 12:46:35.883401 6973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/master-1/wal
uuid: "4fdbce7f7c3d4fda8c562a4fb4072671"
format_stamp: "Formatted at 2026-09-27 12:46:35 on dist-test-slave-lkn4"
server_key: "9d2a09369bc4eb124f11f8091f5f722d"
server_key_iv: "5a798cef9aa445558128f942026d8607"
server_key_version: "encryptionkey@0"
I20260927 12:46:35.883479 6973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:46:35.902452 6973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:46:35.904390 6973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:46:35.908674 6973 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:41239
I20260927 12:46:35.909071 7145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:41239 every 8 connection(s)
I20260927 12:46:35.909327 6973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/master-1/data/info.pb
I20260927 12:46:35.910812 7146 sys_catalog.cc:263] Verifying existing consensus state
I20260927 12:46:35.911751 7146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fdbce7f7c3d4fda8c562a4fb4072671: Bootstrap starting.
I20260927 12:46:35.920204 7146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fdbce7f7c3d4fda8c562a4fb4072671: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 12:46:35.920512 7146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fdbce7f7c3d4fda8c562a4fb4072671: Bootstrap complete.
I20260927 12:46:35.921272 7146 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4fdbce7f7c3d4fda8c562a4fb4072671 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "205c8e3f55144799958e82a465ac9b66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40735 } } peers { permanent_uuid: "4fdbce7f7c3d4fda8c562a4fb4072671" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41239 } attrs { promote: true } }
I20260927 12:46:35.921389 7146 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4fdbce7f7c3d4fda8c562a4fb4072671 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4fdbce7f7c3d4fda8c562a4fb4072671, State: Initialized, Role: LEARNER
I20260927 12:46:35.921489 7146 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fdbce7f7c3d4fda8c562a4fb4072671 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "205c8e3f55144799958e82a465ac9b66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40735 } } peers { permanent_uuid: "4fdbce7f7c3d4fda8c562a4fb4072671" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41239 } attrs { promote: true } }
I20260927 12:46:35.921808 7149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fdbce7f7c3d4fda8c562a4fb4072671 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "205c8e3f55144799958e82a465ac9b66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40735 } } peers { permanent_uuid: "4fdbce7f7c3d4fda8c562a4fb4072671" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41239 } attrs { promote: true } } }
I20260927 12:46:35.921959 7146 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fdbce7f7c3d4fda8c562a4fb4072671 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:46:35.921978 7149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fdbce7f7c3d4fda8c562a4fb4072671 [sys.catalog]: This master's current role is: LEARNER
I20260927 12:46:35.924626 7160 catalog_manager.cc:1259] Loaded cluster ID: ee03576c83fa435283af71c4b0841e2f
I20260927 12:46:35.924746 7160 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4fdbce7f7c3d4fda8c562a4fb4072671: loading cluster ID for follower catalog manager: success
I20260927 12:46:35.926424 7160 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4fdbce7f7c3d4fda8c562a4fb4072671: acquiring CA information for follower catalog manager: success
I20260927 12:46:35.927484 7160 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4fdbce7f7c3d4fda8c562a4fb4072671: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 12:46:36.007844 7082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:46:36.008144 7082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:46:36.008189 7082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:46:36.012145 7082 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:46:36.012212 7082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:46:36.012233 7082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:46:36.012249 7082 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:46:36.012264 7082 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:46:36.017629 7082 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40735,127.0.98.61:41239,127.0.98.60:45213
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7082
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:46:36.018769 7082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:46:36.019872 7082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:46:36.028200 7165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:36.028226 7168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:36.028512 7082 server_base.cc:1034] running on GCE node
W20260927 12:46:36.028226 7166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:36.029245 7082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:46:36.030411 7082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:46:36.031611 7082 hybrid_clock.cc:648] HybridClock initialized: now 1790513196031598 us; error 29 us; skew 500 ppm
Sep 27 12:46:36 dist-test-slave-lkn4 krb5kdc[6464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513196, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:46:36.035175 7082 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260927 12:46:36.036239 7082 webserver.cc:460] Webserver started at http://127.0.98.60:43967/ using document root <none> and password file <none>
I20260927 12:46:36.036841 7082 fs_manager.cc:359] Metadata directory not provided
I20260927 12:46:36.036906 7082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:46:36.037101 7082 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:46:36.039316 7082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/master-2/data/instance:
uuid: "70b88e0990ca471d93129ebffc98d9cf"
format_stamp: "Formatted at 2026-09-27 12:46:36 on dist-test-slave-lkn4"
server_key: "645c6a5b766f657bfdec010decdd8802"
server_key_iv: "6249e053229f084099a184fd4f9ef76e"
server_key_version: "encryptionkey@0"
I20260927 12:46:36.039855 7082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/master-2/wal/instance:
uuid: "70b88e0990ca471d93129ebffc98d9cf"
format_stamp: "Formatted at 2026-09-27 12:46:36 on dist-test-slave-lkn4"
server_key: "645c6a5b766f657bfdec010decdd8802"
server_key_iv: "6249e053229f084099a184fd4f9ef76e"
server_key_version: "encryptionkey@0"
I20260927 12:46:36.043697 7082 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 12:46:36.046134 7175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:36.047498 7082 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:46:36.047621 7082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/master-2/wal
uuid: "70b88e0990ca471d93129ebffc98d9cf"
format_stamp: "Formatted at 2026-09-27 12:46:36 on dist-test-slave-lkn4"
server_key: "645c6a5b766f657bfdec010decdd8802"
server_key_iv: "6249e053229f084099a184fd4f9ef76e"
server_key_version: "encryptionkey@0"
I20260927 12:46:36.047734 7082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:46:36.066531 7082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:46:36.069813 7082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:46:36.069968 7082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:46:36.076833 7082 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:45213
I20260927 12:46:36.077092 7227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:45213 every 8 connection(s)
I20260927 12:46:36.077803 7082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/master-2/data/info.pb
I20260927 12:46:36.080643 7228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:46:36.080770 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 7082
I20260927 12:46:36.086431 7228 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40735 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 12:46:36 dist-test-slave-lkn4 krb5kdc[6464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513194, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 27 12:46:36 dist-test-slave-lkn4 krb5kdc[6464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513196, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 12:46:36.095749 7230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:36.082357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40244 (local address 127.0.98.60:45213)
0927 12:46:36.082757 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:36.082777 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:36.082819 (+ 42us) server_negotiation.cc:407] Connection header received
0927 12:46:36.083701 (+ 882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:36.083726 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:36.084152 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:36.084574 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:36.085491 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.086650 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.087520 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.088152 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.088683 (+ 531us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:36.091003 (+ 2320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:36.091008 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:36.091017 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:36.091042 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:36.093023 (+ 1981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:36.093413 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:36.093417 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:36.093422 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:36.093523 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:36.093790 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:36.093792 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:36.093793 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:36.094218 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:36.094406 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:36.094633 (+ 227us) server_negotiation.cc:299] Negotiation successful
0927 12:46:36.094906 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":118,"threads_started":1}
I20260927 12:46:36.097208 7231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:36.088025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40735 (local address 127.0.0.1:59120)
0927 12:46:36.088379 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:36.088415 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:36.088523 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:36.089024 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:36.089028 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:36.089340 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:36.089452 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:36.089457 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.090426 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.090429 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:36.091095 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:36.091104 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.091610 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.091614 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:36.091717 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:36.092236 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:36.092248 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:36.094251 (+ 2003us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:36.095798 (+ 1547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:36.095805 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:36.095811 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:36.096064 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:36.096315 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:36.096318 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:36.096320 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:36.096396 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:36.096747 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:36.096751 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:36.096838 (+ 87us) client_negotiation.cc:769] Sending connection context
0927 12:46:36.096946 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 12:46:36.097079 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":124,"threads_started":1}
I20260927 12:46:36.097440 6548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:36.088144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59120 (local address 127.0.98.62:40735)
0927 12:46:36.088272 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:36.088277 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:36.088486 (+ 209us) server_negotiation.cc:407] Connection header received
0927 12:46:36.088612 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:36.088616 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:36.088674 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:36.088738 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:36.089571 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.090313 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.091213 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.091507 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.091611 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:36.094395 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:36.094399 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:36.094401 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:36.094426 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:36.095700 (+ 1274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:36.096155 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:36.096158 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:36.096160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:36.096239 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:36.096469 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:36.096472 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:36.096474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:36.096664 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:36.096757 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:36.097013 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 12:46:36.097121 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 12:46:36.098248 7228 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41239 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 12:46:36 dist-test-slave-lkn4 krb5kdc[6464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513196, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 12:46:36.105536 7231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:36.098633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41239 (local address 127.0.0.1:48080)
0927 12:46:36.098733 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:36.098746 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:36.098890 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:36.099297 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:36.099299 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:36.099423 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:36.099507 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:36.099512 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.100378 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.100381 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:36.100963 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:36.100971 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.101439 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.101442 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:36.101513 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:36.101865 (+ 352us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:36.101874 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:36.103030 (+ 1156us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:36.104426 (+ 1396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:36.104428 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:36.104429 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:36.104593 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:36.104830 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:36.104831 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:36.104833 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:36.104906 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:36.105184 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:36.105186 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:36.105251 (+ 65us) client_negotiation.cc:769] Sending connection context
0927 12:46:36.105323 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 12:46:36.105417 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260927 12:46:36.105592 7232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:36.098809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48080 (local address 127.0.98.61:41239)
0927 12:46:36.099037 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:36.099040 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:36.099052 (+ 12us) server_negotiation.cc:407] Connection header received
0927 12:46:36.099104 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:36.099107 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:36.099143 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:36.099261 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:36.099600 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.100279 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.101073 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.101334 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.101411 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:36.103154 (+ 1743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:36.103157 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:36.103159 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:36.103179 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:36.104334 (+ 1155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:36.104657 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:36.104660 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:36.104661 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:36.104725 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:36.104970 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:36.104972 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:36.104973 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:36.105123 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:36.105191 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:36.105365 (+ 174us) server_negotiation.cc:299] Negotiation successful
0927 12:46:36.105458 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":84,"threads_started":1}
I20260927 12:46:36.106292 7228 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45213 } has no permanent_uuid. Determining permanent_uuid...
I20260927 12:46:36.108233 6700 heartbeater.cc:503] Master 127.0.98.62:40735 was elected leader, sending a full tablet report...
Sep 27 12:46:36 dist-test-slave-lkn4 krb5kdc[6464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513196, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 12:46:36.114300 7231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:36.106849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45213 (local address 127.0.0.1:40246)
0927 12:46:36.106931 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:36.106941 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:36.107002 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:36.107194 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:36.107196 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:36.107332 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:36.107392 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:36.107396 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.108437 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.108440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:36.109001 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:36.109009 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.109419 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.109424 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:36.109510 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:36.110024 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:36.110070 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:36.111219 (+ 1149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:36.112870 (+ 1651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:36.112874 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:36.112876 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:36.113074 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:36.113386 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:36.113388 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:36.113390 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:36.113468 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:36.113833 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:36.113836 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:36.113910 (+ 74us) client_negotiation.cc:769] Sending connection context
0927 12:46:36.113991 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 12:46:36.114169 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260927 12:46:36.114391 7230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:36.106956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40246 (local address 127.0.98.60:45213)
0927 12:46:36.107049 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:36.107053 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:36.107064 (+ 11us) server_negotiation.cc:407] Connection header received
0927 12:46:36.107090 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:36.107092 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:36.107118 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:36.107176 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:36.107483 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.108351 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.109094 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.109317 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.109410 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:36.111332 (+ 1922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:36.111336 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:36.111338 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:36.111358 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:36.112619 (+ 1261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:36.113192 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:36.113195 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:36.113197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:36.113274 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:36.113556 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:36.113558 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:36.113560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:36.113743 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:36.113821 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:36.114078 (+ 257us) server_negotiation.cc:299] Negotiation successful
0927 12:46:36.114250 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":38}
I20260927 12:46:36.117131 7228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf: Bootstrap starting.
I20260927 12:46:36.119417 7228 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf: Neither blocks nor log segments found. Creating new log.
I20260927 12:46:36.120111 7228 log.cc:824] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf: Log is configured to *not* fsync() on all Append() calls
I20260927 12:46:36.122012 7228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf: No bootstrap required, opened a new log
I20260927 12:46:36.125092 7228 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205c8e3f55144799958e82a465ac9b66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40735 } } peers { permanent_uuid: "4fdbce7f7c3d4fda8c562a4fb4072671" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41239 } } peers { permanent_uuid: "70b88e0990ca471d93129ebffc98d9cf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45213 } }
I20260927 12:46:36.125293 7228 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:46:36.125329 7228 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70b88e0990ca471d93129ebffc98d9cf, State: Initialized, Role: FOLLOWER
I20260927 12:46:36.125756 7228 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205c8e3f55144799958e82a465ac9b66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40735 } } peers { permanent_uuid: "4fdbce7f7c3d4fda8c562a4fb4072671" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41239 } } peers { permanent_uuid: "70b88e0990ca471d93129ebffc98d9cf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45213 } }
I20260927 12:46:36.126250 7234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205c8e3f55144799958e82a465ac9b66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40735 } } peers { permanent_uuid: "4fdbce7f7c3d4fda8c562a4fb4072671" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41239 } } peers { permanent_uuid: "70b88e0990ca471d93129ebffc98d9cf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45213 } } }
I20260927 12:46:36.126363 7234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:46:36.126945 7228 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf [sys.catalog]: configured and running, proceeding with master startup.
W20260927 12:46:36.131330 7245 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 12:46:36.131408 7245 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 12:46:36.134876 7082 master_runner.cc:428] Detected that this master 70b88e0990ca471d93129ebffc98d9cf is joining an existing cluster
I20260927 12:46:36.134932 7082 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:45213
I20260927 12:46:36.144883 7230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:36.137874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40250 (local address 127.0.98.60:45213)
0927 12:46:36.137952 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:36.137956 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:36.138015 (+ 59us) server_negotiation.cc:407] Connection header received
0927 12:46:36.138189 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:36.138192 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:36.138225 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:36.138264 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:36.138953 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.139627 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.140734 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.141016 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.141076 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:36.142287 (+ 1211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:36.142290 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:36.142293 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:36.142312 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:36.143521 (+ 1209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:36.143868 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:36.143872 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:36.143874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:36.143937 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:36.144152 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:36.144155 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:36.144157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:36.144318 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:36.144398 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:36.144606 (+ 208us) server_negotiation.cc:299] Negotiation successful
0927 12:46:36.144742 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":34}
I20260927 12:46:36.144897 7253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:36.137643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45213 (local address 127.0.0.1:40250)
0927 12:46:36.137952 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:36.137964 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:36.138108 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:36.138401 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:36.138404 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:36.138724 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:36.138849 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:36.138856 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.139759 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.139761 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:36.140628 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:36.140637 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.141202 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.141205 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:36.141296 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:36.141810 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:36.141820 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:36.142181 (+ 361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:36.143612 (+ 1431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:36.143615 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:36.143617 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:36.143794 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:36.144010 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:36.144013 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:36.144015 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:36.144086 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:36.144398 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:36.144401 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:36.144479 (+ 78us) client_negotiation.cc:769] Sending connection context
0927 12:46:36.144558 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 12:46:36.144692 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"spinlock_wait_cycles":7936,"thread_start_us":67,"threads_started":1}
I20260927 12:46:36.148942 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 6571
I20260927 12:46:36.155858 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:43975
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40735
--webserver_port=42583
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:40735,127.0.98.61:41239
--tserver_master_addrs=127.0.98.62:40735,127.0.98.61:41239,127.0.98.60:45213 with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 12:46:36.159374 7232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:36.137338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48090 (local address 127.0.98.61:41239)
0927 12:46:36.137453 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:36.137457 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:36.137926 (+ 469us) server_negotiation.cc:407] Connection header received
0927 12:46:36.138136 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:36.138140 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:36.138187 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:36.138239 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:36.138991 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.139801 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.141034 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.141370 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.141445 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:36.155404 (+ 13959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:36.155410 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:36.155412 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:36.155443 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:36.157233 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:36.157901 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:36.157904 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:36.157906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:36.157995 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:36.158406 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:36.158410 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:36.158412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:36.158638 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:36.158744 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:36.159078 (+ 334us) server_negotiation.cc:299] Negotiation successful
0927 12:46:36.159206 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 12:46:36.159751 6548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:36.137682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59134 (local address 127.0.98.62:40735)
0927 12:46:36.137785 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:36.137789 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:36.137803 (+ 14us) server_negotiation.cc:407] Connection header received
0927 12:46:36.137844 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:36.137847 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:36.137895 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:36.137969 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:36.138538 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.139406 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.141033 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.141334 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.141405 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:36.155404 (+ 13999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:36.155410 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:36.155412 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:36.155441 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:36.157975 (+ 2534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:36.158504 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:36.158508 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:36.158509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:36.158598 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:36.158893 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:36.158896 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:36.158898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:36.159101 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:36.159166 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:36.159480 (+ 314us) server_negotiation.cc:299] Negotiation successful
0927 12:46:36.159596 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 12:46:36.159996 7251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:36.137307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41239 (local address 127.0.0.1:48090)
0927 12:46:36.137840 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:36.137852 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:36.137983 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:36.138374 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:36.138376 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:36.138734 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:36.138861 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:36.138868 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.139960 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.139963 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:36.140902 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:36.140913 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.141589 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.141592 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:36.141694 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:36.154661 (+ 12967us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:36.154673 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:36.155216 (+ 543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:36.157392 (+ 2176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:36.157396 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:36.157398 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:36.157652 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:36.158131 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:36.158134 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:36.158136 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:36.158301 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:36.158814 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:36.158817 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:36.158912 (+ 95us) client_negotiation.cc:769] Sending connection context
0927 12:46:36.159567 (+ 655us) client_negotiation.cc:240] Negotiation successful
0927 12:46:36.159733 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"mutex_wait_us":29,"spinlock_wait_cycles":4992,"thread_start_us":94,"threads_started":1}
I20260927 12:46:36.160218 7252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:36.137239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40735 (local address 127.0.0.1:59134)
0927 12:46:36.137567 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:36.137581 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:36.137658 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:36.138007 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:36.138010 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:36.138247 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:36.138414 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:36.138421 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.139534 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.139537 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:36.140922 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:36.140930 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.141547 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.141550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:36.141631 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:36.154577 (+ 12946us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:36.154596 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:36.155216 (+ 620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:36.158129 (+ 2913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:36.158132 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:36.158135 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:36.158409 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:36.158694 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:36.158697 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:36.158698 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:36.158795 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:36.159230 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:36.159233 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:36.159316 (+ 83us) client_negotiation.cc:769] Sending connection context
0927 12:46:36.159902 (+ 586us) client_negotiation.cc:240] Negotiation successful
0927 12:46:36.160044 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":4,"spinlock_wait_cycles":5376,"thread_start_us":104,"threads_started":1}
Sep 27 12:46:36 dist-test-slave-lkn4 krb5kdc[6464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513194, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 12:46:36.173254 7066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:36.164260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45213 (local address 127.0.0.1:40266)
0927 12:46:36.164474 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:36.164487 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:36.164555 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:36.165112 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:36.165115 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:36.165298 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:36.165402 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:36.165409 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.166504 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.166509 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:36.167366 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:36.167386 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.167925 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.167929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:36.168025 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:36.168473 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:36.168487 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:36.169942 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:36.171746 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:36.171750 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:36.171753 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:36.172019 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:36.172270 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:36.172273 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:36.172275 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:36.172375 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:36.172752 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:36.172755 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:36.172850 (+ 95us) client_negotiation.cc:769] Sending connection context
0927 12:46:36.172951 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 12:46:36.173085 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147}
I20260927 12:46:36.173261 7230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:36.164689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40266 (local address 127.0.98.60:45213)
0927 12:46:36.164820 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:36.164825 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:36.164839 (+ 14us) server_negotiation.cc:407] Connection header received
0927 12:46:36.164927 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:36.164930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:36.164984 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:36.165060 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:36.165562 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.166399 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.167487 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.167829 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.167921 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:36.170085 (+ 2164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:36.170089 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:36.170091 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:36.170115 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:36.171628 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:36.172104 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:36.172107 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:36.172109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:36.172186 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:36.172455 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:36.172458 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:36.172460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:36.172652 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:36.172745 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:36.173011 (+ 266us) server_negotiation.cc:299] Negotiation successful
0927 12:46:36.173115 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 12:46:36.174539 6509 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:45213: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "70b88e0990ca471d93129ebffc98d9cf"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 45213
}
attrs {
promote: true
}
}
dest_uuid: "205c8e3f55144799958e82a465ac9b66"
cas_config_opid_index: 5
I20260927 12:46:36.175159 6509 consensus_queue.cc:237] T 00000000000000000000000000000000 P 205c8e3f55144799958e82a465ac9b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "205c8e3f55144799958e82a465ac9b66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40735 } } peers { permanent_uuid: "4fdbce7f7c3d4fda8c562a4fb4072671" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41239 } attrs { promote: true } } peers { permanent_uuid: "70b88e0990ca471d93129ebffc98d9cf" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45213 } attrs { promote: true } }
I20260927 12:46:36.176162 7070 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 205c8e3f55144799958e82a465ac9b66 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 70b88e0990ca471d93129ebffc98d9cf (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "205c8e3f55144799958e82a465ac9b66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40735 } } peers { permanent_uuid: "4fdbce7f7c3d4fda8c562a4fb4072671" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41239 } attrs { promote: true } } peers { permanent_uuid: "70b88e0990ca471d93129ebffc98d9cf" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45213 } attrs { promote: true } } }
I20260927 12:46:36.177286 7202 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:46:36.177565 7070 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:45213
I20260927 12:46:36.177721 7070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 205c8e3f55144799958e82a465ac9b66 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "205c8e3f55144799958e82a465ac9b66" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "205c8e3f55144799958e82a465ac9b66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40735 } } peers { permanent_uuid: "4fdbce7f7c3d4fda8c562a4fb4072671" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41239 } attrs { promote: true } } peers { permanent_uuid: "70b88e0990ca471d93129ebffc98d9cf" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45213 } attrs { promote: true } } }
I20260927 12:46:36.177819 7070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 205c8e3f55144799958e82a465ac9b66 [sys.catalog]: This master's current role is: LEADER
I20260927 12:46:36.178815 7202 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf [term 1 FOLLOWER]: Refusing update from remote peer 205c8e3f55144799958e82a465ac9b66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 12:46:36.179637 7068 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 205c8e3f55144799958e82a465ac9b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70b88e0990ca471d93129ebffc98d9cf" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45213 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:46:36.179767 7082 master.cc:537] Master@127.0.98.60:45213 shutting down...
W20260927 12:46:36.181401 6493 consensus_peers.cc:487] T 00000000000000000000000000000000 P 205c8e3f55144799958e82a465ac9b66 -> Peer 70b88e0990ca471d93129ebffc98d9cf (127.0.98.60:45213): Couldn't send request to peer 70b88e0990ca471d93129ebffc98d9cf. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 12:46:36.185451 7082 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:46:36.185846 7082 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:46:36.186023 7082 tablet_replica.cc:330] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf: stopping tablet replica
I20260927 12:46:36.190204 7232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:36.176637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48098 (local address 127.0.98.61:41239)
0927 12:46:36.181415 (+ 4778us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:36.181420 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:36.184489 (+ 3069us) server_negotiation.cc:407] Connection header received
0927 12:46:36.184556 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:36.184560 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:36.184630 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:36.184724 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:46:36.185430 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.186251 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.188051 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.188936 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.189024 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:36.189194 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:36.189964 (+ 770us) server_negotiation.cc:299] Negotiation successful
0927 12:46:36.190069 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4679}
I20260927 12:46:36.189776 7066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:36.176382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41239 (local address 127.0.0.1:48098)
0927 12:46:36.184246 (+ 7864us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:36.184267 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:36.184385 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:36.185117 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:36.185122 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:36.185148 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:46:36.185259 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:36.185267 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.186401 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.186404 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:36.187680 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:36.187693 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.189302 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.189306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:36.189407 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:36.189466 (+ 59us) client_negotiation.cc:769] Sending connection context
0927 12:46:36.189540 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 12:46:36.189622 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7770}
I20260927 12:46:36.191036 7120 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4fdbce7f7c3d4fda8c562a4fb4072671 [term 1 LEARNER]: Refusing update from remote peer 205c8e3f55144799958e82a465ac9b66: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 12:46:36.191151 7082 master.cc:552] Master@127.0.98.60:45213 shutdown complete.
I20260927 12:46:36.191223 7082 master_runner.cc:305] Clearing existing system tablet
I20260927 12:46:36.191521 7068 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 205c8e3f55144799958e82a465ac9b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fdbce7f7c3d4fda8c562a4fb4072671" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41239 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 12:46:36.191881 7082 fs_manager.cc:359] Metadata directory not provided
I20260927 12:46:36.191948 7082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:46:36.193123 7082 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 12:46:36.193522 7120 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4fdbce7f7c3d4fda8c562a4fb4072671 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 70b88e0990ca471d93129ebffc98d9cf (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "205c8e3f55144799958e82a465ac9b66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40735 } } peers { permanent_uuid: "4fdbce7f7c3d4fda8c562a4fb4072671" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41239 } attrs { promote: true } } peers { permanent_uuid: "70b88e0990ca471d93129ebffc98d9cf" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45213 } attrs { promote: true } } }
I20260927 12:46:36.193881 7257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:36.194075 7082 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:46:36.194180 7082 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260927 12:46:36.194207 7082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/master-2/wal
uuid: "70b88e0990ca471d93129ebffc98d9cf"
format_stamp: "Formatted at 2026-09-27 12:46:36 on dist-test-slave-lkn4"
server_key: "645c6a5b766f657bfdec010decdd8802"
server_key_iv: "6249e053229f084099a184fd4f9ef76e"
server_key_version: "encryptionkey@0"
I20260927 12:46:36.195369 7149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fdbce7f7c3d4fda8c562a4fb4072671 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 205c8e3f55144799958e82a465ac9b66. Latest consensus state: current_term: 1 leader_uuid: "205c8e3f55144799958e82a465ac9b66" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "205c8e3f55144799958e82a465ac9b66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40735 } } peers { permanent_uuid: "4fdbce7f7c3d4fda8c562a4fb4072671" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41239 } attrs { promote: true } } peers { permanent_uuid: "70b88e0990ca471d93129ebffc98d9cf" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45213 } attrs { promote: true } } }
I20260927 12:46:36.195482 7149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fdbce7f7c3d4fda8c562a4fb4072671 [sys.catalog]: This master's current role is: LEARNER
I20260927 12:46:36.195560 7149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fdbce7f7c3d4fda8c562a4fb4072671 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "205c8e3f55144799958e82a465ac9b66" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "205c8e3f55144799958e82a465ac9b66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40735 } } peers { permanent_uuid: "4fdbce7f7c3d4fda8c562a4fb4072671" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41239 } attrs { promote: true } } peers { permanent_uuid: "70b88e0990ca471d93129ebffc98d9cf" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45213 } attrs { promote: true } } }
I20260927 12:46:36.195631 7149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fdbce7f7c3d4fda8c562a4fb4072671 [sys.catalog]: This master's current role is: LEARNER
I20260927 12:46:36.196161 7082 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:46:36.198428 7082 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 12:46:36.198518 7082 log.cc:1196] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf: Deleting WAL directory at /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260927 12:46:36.198737 7082 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf: Deleting consensus metadata
I20260927 12:46:36.198905 7082 master_runner.cc:315] Copying system tablet from 127.0.98.62:40735
I20260927 12:46:36.199762 7068 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 205c8e3f55144799958e82a465ac9b66: attempting to promote NON_VOTER 4fdbce7f7c3d4fda8c562a4fb4072671 to VOTER
I20260927 12:46:36.199988 7082 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:40735
I20260927 12:46:36.200268 7068 consensus_queue.cc:237] T 00000000000000000000000000000000 P 205c8e3f55144799958e82a465ac9b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "205c8e3f55144799958e82a465ac9b66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40735 } } peers { permanent_uuid: "4fdbce7f7c3d4fda8c562a4fb4072671" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41239 } attrs { promote: false } } peers { permanent_uuid: "70b88e0990ca471d93129ebffc98d9cf" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45213 } attrs { promote: true } }
I20260927 12:46:36.202242 7120 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4fdbce7f7c3d4fda8c562a4fb4072671 [term 1 LEARNER]: Refusing update from remote peer 205c8e3f55144799958e82a465ac9b66: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 12:46:36.202713 7069 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 205c8e3f55144799958e82a465ac9b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fdbce7f7c3d4fda8c562a4fb4072671" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41239 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 12:46:36.204042 7066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:36.201676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45213 (local address 127.0.0.1:40278)
0927 12:46:36.203890 (+ 2214us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:36.203955 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2139}
I20260927 12:46:36.205509 7066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:36.204769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45213 (local address 127.0.0.1:40286)
0927 12:46:36.205374 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:36.205428 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":411,"threads_started":1}
W20260927 12:46:36.205686 6493 consensus_peers.cc:487] T 00000000000000000000000000000000 P 205c8e3f55144799958e82a465ac9b66 -> Peer 70b88e0990ca471d93129ebffc98d9cf (127.0.98.60:45213): Couldn't send request to peer 70b88e0990ca471d93129ebffc98d9cf. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45213: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 12:46:36.207321 7069 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 205c8e3f55144799958e82a465ac9b66 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 4fdbce7f7c3d4fda8c562a4fb4072671 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "205c8e3f55144799958e82a465ac9b66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40735 } } peers { permanent_uuid: "4fdbce7f7c3d4fda8c562a4fb4072671" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41239 } attrs { promote: false } } peers { permanent_uuid: "70b88e0990ca471d93129ebffc98d9cf" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45213 } attrs { promote: true } } }
I20260927 12:46:36.208889 7069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 205c8e3f55144799958e82a465ac9b66 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "205c8e3f55144799958e82a465ac9b66" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "205c8e3f55144799958e82a465ac9b66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40735 } } peers { permanent_uuid: "4fdbce7f7c3d4fda8c562a4fb4072671" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41239 } attrs { promote: false } } peers { permanent_uuid: "70b88e0990ca471d93129ebffc98d9cf" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45213 } attrs { promote: true } } }
I20260927 12:46:36.209000 7069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 205c8e3f55144799958e82a465ac9b66 [sys.catalog]: This master's current role is: LEADER
I20260927 12:46:36.209141 6548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:36.200657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59144 (local address 127.0.98.62:40735)
0927 12:46:36.200766 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:36.200782 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:36.201227 (+ 445us) server_negotiation.cc:407] Connection header received
0927 12:46:36.201406 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:36.201410 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:36.201455 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:36.201504 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:36.202119 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.202905 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.204043 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.204355 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.204424 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:36.205759 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:36.205763 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:36.205766 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:36.205806 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:36.207390 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:36.207860 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:36.207863 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:36.207865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:36.207947 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:36.208216 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:36.208219 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:36.208221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:36.208420 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:36.208485 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:36.208850 (+ 365us) server_negotiation.cc:299] Negotiation successful
0927 12:46:36.208966 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 12:46:36.209141 7069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 205c8e3f55144799958e82a465ac9b66 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "205c8e3f55144799958e82a465ac9b66" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "205c8e3f55144799958e82a465ac9b66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40735 } } peers { permanent_uuid: "4fdbce7f7c3d4fda8c562a4fb4072671" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41239 } attrs { promote: false } } peers { permanent_uuid: "70b88e0990ca471d93129ebffc98d9cf" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45213 } attrs { promote: true } } }
I20260927 12:46:36.209236 7069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 205c8e3f55144799958e82a465ac9b66 [sys.catalog]: This master's current role is: LEADER
I20260927 12:46:36.209609 7264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:36.200881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40735 (local address 127.0.0.1:59144)
0927 12:46:36.201151 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:36.201164 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:36.201298 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:36.201643 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:36.201645 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:36.201865 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:36.201974 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:36.201981 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.203038 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.203041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:36.203912 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:36.203922 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.204577 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.204580 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:36.204667 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:36.205196 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:36.205209 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:36.205619 (+ 410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:36.207506 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:36.207509 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:36.207511 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:36.207746 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:36.208044 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:36.208047 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:36.208049 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:36.208130 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:36.208554 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:36.208557 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:36.208654 (+ 97us) client_negotiation.cc:769] Sending connection context
0927 12:46:36.209152 (+ 498us) client_negotiation.cc:240] Negotiation successful
0927 12:46:36.209432 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":58,"threads_started":1}
I20260927 12:46:36.210481 6529 tablet_copy_service.cc:140] P 205c8e3f55144799958e82a465ac9b66: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 70b88e0990ca471d93129ebffc98d9cf ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:59144)
I20260927 12:46:36.210574 6529 tablet_copy_service.cc:161] P 205c8e3f55144799958e82a465ac9b66: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 70b88e0990ca471d93129ebffc98d9cf at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:59144: session id = 70b88e0990ca471d93129ebffc98d9cf-00000000000000000000000000000000
I20260927 12:46:36.210495 7119 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4fdbce7f7c3d4fda8c562a4fb4072671 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 4fdbce7f7c3d4fda8c562a4fb4072671 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "205c8e3f55144799958e82a465ac9b66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40735 } } peers { permanent_uuid: "4fdbce7f7c3d4fda8c562a4fb4072671" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41239 } attrs { promote: false } } peers { permanent_uuid: "70b88e0990ca471d93129ebffc98d9cf" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45213 } attrs { promote: true } } }
I20260927 12:46:36.211354 6529 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 205c8e3f55144799958e82a465ac9b66: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 12:46:36.212818 7082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 12:46:36.214179 7258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fdbce7f7c3d4fda8c562a4fb4072671 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "205c8e3f55144799958e82a465ac9b66" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "205c8e3f55144799958e82a465ac9b66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40735 } } peers { permanent_uuid: "4fdbce7f7c3d4fda8c562a4fb4072671" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41239 } attrs { promote: false } } peers { permanent_uuid: "70b88e0990ca471d93129ebffc98d9cf" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45213 } attrs { promote: true } } }
I20260927 12:46:36.214403 7258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fdbce7f7c3d4fda8c562a4fb4072671 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:46:36.215189 7082 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf: tablet copy: Starting download of 0 data blocks...
I20260927 12:46:36.215363 7082 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf: tablet copy: Starting download of 1 WAL segments...
I20260927 12:46:36.216830 7082 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 12:46:36.218598 6529 tablet_copy_service.cc:342] P 205c8e3f55144799958e82a465ac9b66: Request end of tablet copy session 70b88e0990ca471d93129ebffc98d9cf-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:59144
I20260927 12:46:36.218688 6529 tablet_copy_service.cc:433] P 205c8e3f55144799958e82a465ac9b66: ending tablet copy session 70b88e0990ca471d93129ebffc98d9cf-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 70b88e0990ca471d93129ebffc98d9cf
I20260927 12:46:36.226301 7082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:46:36.228236 7266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:36.228255 7269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:36.228829 7082 server_base.cc:1034] running on GCE node
W20260927 12:46:36.228858 7267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:36.229156 7082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:46:36.229453 7082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:46:36.230584 7082 hybrid_clock.cc:648] HybridClock initialized: now 1790513196230572 us; error 31 us; skew 500 ppm
Sep 27 12:46:36 dist-test-slave-lkn4 krb5kdc[6464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513196, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:46:36.232151 7082 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260927 12:46:36.232621 7082 webserver.cc:460] Webserver started at http://127.0.98.60:35123/ using document root <none> and password file <none>
I20260927 12:46:36.232803 7082 fs_manager.cc:359] Metadata directory not provided
I20260927 12:46:36.232843 7082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:46:36.234148 7082 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 12:46:36.235070 7276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:36.235374 7082 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 12:46:36.235451 7082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/master-2/wal
uuid: "70b88e0990ca471d93129ebffc98d9cf"
format_stamp: "Formatted at 2026-09-27 12:46:36 on dist-test-slave-lkn4"
server_key: "645c6a5b766f657bfdec010decdd8802"
server_key_iv: "6249e053229f084099a184fd4f9ef76e"
server_key_version: "encryptionkey@0"
I20260927 12:46:36.235530 7082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:46:36.261502 7082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:46:36.263926 7082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:46:36.269088 7082 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:45213
I20260927 12:46:36.269121 7328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:45213 every 8 connection(s)
I20260927 12:46:36.269662 7082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/master-2/data/info.pb
I20260927 12:46:36.271497 7329 sys_catalog.cc:263] Verifying existing consensus state
I20260927 12:46:36.272681 7329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf: Bootstrap starting.
I20260927 12:46:36.284114 7329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 12:46:36.284600 7329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf: Bootstrap complete.
I20260927 12:46:36.285339 7329 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "205c8e3f55144799958e82a465ac9b66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40735 } } peers { permanent_uuid: "4fdbce7f7c3d4fda8c562a4fb4072671" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41239 } attrs { promote: false } } peers { permanent_uuid: "70b88e0990ca471d93129ebffc98d9cf" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45213 } attrs { promote: true } }
I20260927 12:46:36.285586 7329 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 70b88e0990ca471d93129ebffc98d9cf, State: Initialized, Role: LEARNER
I20260927 12:46:36.285727 7329 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "205c8e3f55144799958e82a465ac9b66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40735 } } peers { permanent_uuid: "4fdbce7f7c3d4fda8c562a4fb4072671" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41239 } attrs { promote: false } } peers { permanent_uuid: "70b88e0990ca471d93129ebffc98d9cf" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45213 } attrs { promote: true } }
I20260927 12:46:36.286197 7332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "205c8e3f55144799958e82a465ac9b66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40735 } } peers { permanent_uuid: "4fdbce7f7c3d4fda8c562a4fb4072671" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41239 } attrs { promote: false } } peers { permanent_uuid: "70b88e0990ca471d93129ebffc98d9cf" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45213 } attrs { promote: true } } }
I20260927 12:46:36.286331 7332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf [sys.catalog]: This master's current role is: LEARNER
I20260927 12:46:36.286293 7329 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:46:36.290205 7343 catalog_manager.cc:1259] Loaded cluster ID: ee03576c83fa435283af71c4b0841e2f
I20260927 12:46:36.290262 7343 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf: loading cluster ID for follower catalog manager: success
I20260927 12:46:36.292474 7343 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf: acquiring CA information for follower catalog manager: success
I20260927 12:46:36.294230 7343 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 12:46:36.298550 7254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:46:36.298795 7254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:46:36.298838 7254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:46:36.302908 7254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:46:36.302974 7254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:46:36.303054 7254 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:46:36.308408 7254 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:43975
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=42583
--tserver_master_addrs=127.0.98.62:40735,127.0.98.61:41239,127.0.98.60:45213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7254
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:46:36.309458 7254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:46:36.310508 7254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 12:46:36.312824 6834 heartbeater.cc:503] Master 127.0.98.62:40735 was elected leader, sending a full tablet report...
I20260927 12:46:36.320273 7254 server_base.cc:1034] running on GCE node
W20260927 12:46:36.320218 7348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:36.320232 7351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:36.320222 7349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:36.320806 7254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:46:36.321405 7254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:46:36.322570 7254 hybrid_clock.cc:648] HybridClock initialized: now 1790513196322538 us; error 42 us; skew 500 ppm
Sep 27 12:46:36 dist-test-slave-lkn4 krb5kdc[6464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513196, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:46:36.325168 7254 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 12:46:36.325995 7254 webserver.cc:460] Webserver started at http://127.0.98.1:42583/ using document root <none> and password file <none>
I20260927 12:46:36.326556 7254 fs_manager.cc:359] Metadata directory not provided
I20260927 12:46:36.326638 7254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:46:36.330448 7254 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260927 12:46:36.332407 7358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:36.333670 7254 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:46:36.333814 7254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-0/wal
uuid: "9387b2fa57714be0aa7b8e69be342b8e"
format_stamp: "Formatted at 2026-09-27 12:46:35 on dist-test-slave-lkn4"
server_key: "80ca6b0de6bcbc890271b1c51712afab"
server_key_iv: "2a98600608c2265f5b6c0c30e12c1949"
server_key_version: "encryptionkey@0"
I20260927 12:46:36.334396 7254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:46:36.361667 7254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:46:36.365119 7254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:46:36.365309 7254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:46:36.365957 7254 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:46:36.366895 7254 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:46:36.366941 7254 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:36.366995 7254 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:46:36.367024 7254 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:36.376324 7254 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43975
I20260927 12:46:36.376353 7471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43975 every 8 connection(s)
I20260927 12:46:36.377373 7254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:46:36.384956 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 7254
I20260927 12:46:36.385142 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 6705
Sep 27 12:46:36 dist-test-slave-lkn4 krb5kdc[6464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513196, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 27 12:46:36 dist-test-slave-lkn4 krb5kdc[6464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513196, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 27 12:46:36 dist-test-slave-lkn4 krb5kdc[6464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513196, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 12:46:36.392086 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:41427
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40735
--webserver_port=40097
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:40735,127.0.98.61:41239
--tserver_master_addrs=127.0.98.62:40735,127.0.98.61:41239,127.0.98.60:45213 with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 12:46:36.393862 7232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:36.380587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56385 (local address 127.0.98.61:41239)
0927 12:46:36.380699 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:36.380702 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:36.380770 (+ 68us) server_negotiation.cc:407] Connection header received
0927 12:46:36.381283 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:36.381286 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:36.381343 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:36.381410 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:36.382768 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.383586 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.386346 (+ 2760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.386680 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.387029 (+ 349us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:36.390131 (+ 3102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:36.390134 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:36.390137 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:36.390169 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:36.391565 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:36.392053 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:36.392057 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:36.392059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:36.392159 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:36.392566 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:36.392569 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:36.392571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:36.392782 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:36.392860 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:36.393588 (+ 728us) server_negotiation.cc:299] Negotiation successful
0927 12:46:36.393708 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 12:46:36.393944 6548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:36.379097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34515 (local address 127.0.98.62:40735)
0927 12:46:36.379225 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:36.379230 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:36.380261 (+ 1031us) server_negotiation.cc:407] Connection header received
0927 12:46:36.381283 (+ 1022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:36.381286 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:36.381343 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:36.381395 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:36.382805 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.383594 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.386451 (+ 2857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.386783 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.386861 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:36.389553 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:36.389557 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:36.389560 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:36.389590 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:36.391971 (+ 2381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:36.392695 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:36.392698 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:36.392700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:36.392787 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:36.393052 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:36.393054 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:36.393056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:36.393261 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:36.393331 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:36.393674 (+ 343us) server_negotiation.cc:299] Negotiation successful
0927 12:46:36.393791 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 12:46:36.394668 7476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:36.379091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42373 (local address 127.0.98.60:45213)
0927 12:46:36.379368 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:36.379386 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:36.380509 (+ 1123us) server_negotiation.cc:407] Connection header received
0927 12:46:36.381351 (+ 842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:36.381354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:36.381403 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:36.381471 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:36.382787 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.383648 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.385677 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.386012 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.386126 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:36.389799 (+ 3673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:36.389803 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:36.389806 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:36.389834 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:36.391354 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:36.392053 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:36.392057 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:36.392059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:36.392155 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:36.392564 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:36.392567 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:36.392568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:36.392782 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:36.392862 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:36.394370 (+ 1508us) server_negotiation.cc:299] Negotiation successful
0927 12:46:36.394509 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":77,"threads_started":1}
I20260927 12:46:36.395222 7479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:36.380460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41239 (local address 127.0.98.1:56385)
0927 12:46:36.380694 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:36.380706 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:36.381062 (+ 356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:36.381583 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:36.381590 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:36.382069 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:36.382636 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:36.382649 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.383959 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.383962 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:36.386228 (+ 2266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:36.386237 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.387199 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.387203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:36.387538 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:36.387971 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:36.387985 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:36.389959 (+ 1974us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:36.391688 (+ 1729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:36.391691 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:36.391693 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:36.391958 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:36.392260 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:36.392263 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:36.392265 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:36.392464 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:36.392912 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:36.392919 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:36.393304 (+ 385us) client_negotiation.cc:769] Sending connection context
0927 12:46:36.394075 (+ 771us) client_negotiation.cc:240] Negotiation successful
0927 12:46:36.394214 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":255,"spinlock_wait_cycles":9216,"thread_start_us":80,"threads_started":1}
I20260927 12:46:36.395233 7478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:36.379521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45213 (local address 127.0.98.1:42373)
0927 12:46:36.380432 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:36.380443 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:36.381059 (+ 616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:36.381584 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:36.381591 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:36.382069 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:36.382644 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:36.382654 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.383766 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.383769 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:36.385266 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:36.385276 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.386298 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.386302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:36.386484 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:36.387063 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:36.387085 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:36.389667 (+ 2582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:36.391483 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:36.391489 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:36.391509 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:36.391948 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:36.392304 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:36.392307 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:36.392308 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:36.392464 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:36.392922 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:36.392925 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:36.393308 (+ 383us) client_negotiation.cc:769] Sending connection context
0927 12:46:36.393570 (+ 262us) client_negotiation.cc:240] Negotiation successful
0927 12:46:36.393931 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":840,"mutex_wait_us":249,"spinlock_wait_cycles":82432,"thread_start_us":147,"threads_started":1}
I20260927 12:46:36.395915 7477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:36.379521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40735 (local address 127.0.98.1:34515)
0927 12:46:36.380062 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:36.380120 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:36.381057 (+ 937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:36.381584 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:36.381591 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:36.382069 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:36.382652 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:36.382658 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.383724 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.383727 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:36.386311 (+ 2584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:36.386322 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.387010 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.387013 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:36.387101 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:36.387542 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:36.387555 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:36.389361 (+ 1806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:36.392093 (+ 2732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:36.392096 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:36.392098 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:36.392584 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:36.392888 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:36.392890 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:36.392892 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:36.392966 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:36.393404 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:36.393407 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:36.393498 (+ 91us) client_negotiation.cc:769] Sending connection context
0927 12:46:36.394506 (+ 1008us) client_negotiation.cc:240] Negotiation successful
0927 12:46:36.394643 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"mutex_wait_us":256,"spinlock_wait_cycles":70272,"thread_start_us":619,"threads_started":1}
I20260927 12:46:36.396611 7474 heartbeater.cc:347] Connected to a master server at 127.0.98.61:41239
I20260927 12:46:36.396631 7473 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40735
I20260927 12:46:36.396859 7474 heartbeater.cc:464] Registering TS with master...
I20260927 12:46:36.396858 7473 heartbeater.cc:464] Registering TS with master...
I20260927 12:46:36.397401 7473 heartbeater.cc:511] Master 127.0.98.62:40735 requested a full tablet report, sending...
I20260927 12:46:36.397430 7472 heartbeater.cc:347] Connected to a master server at 127.0.98.60:45213
I20260927 12:46:36.397403 7474 heartbeater.cc:511] Master 127.0.98.61:41239 requested a full tablet report, sending...
I20260927 12:46:36.397498 7472 heartbeater.cc:464] Registering TS with master...
I20260927 12:46:36.397627 7472 heartbeater.cc:511] Master 127.0.98.60:45213 requested a full tablet report, sending...
I20260927 12:46:36.398636 6509 ts_manager.cc:194] Re-registered known tserver with Master: 9387b2fa57714be0aa7b8e69be342b8e (127.0.98.1:43975)
I20260927 12:46:36.399047 7285 ts_manager.cc:194] Registered new tserver with Master: 9387b2fa57714be0aa7b8e69be342b8e (127.0.98.1:43975)
I20260927 12:46:36.399047 7109 ts_manager.cc:194] Registered new tserver with Master: 9387b2fa57714be0aa7b8e69be342b8e (127.0.98.1:43975)
I20260927 12:46:36.399399 6509 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34515
W20260927 12:46:36.505988 7480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:46:36.506255 7480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:46:36.506299 7480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:46:36.510290 7480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:46:36.510345 7480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:46:36.510403 7480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:46:36.515580 7480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:41427
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=40097
--tserver_master_addrs=127.0.98.62:40735,127.0.98.61:41239,127.0.98.60:45213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:46:36.516624 7480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:46:36.517609 7480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 12:46:36.524372 6968 heartbeater.cc:503] Master 127.0.98.62:40735 was elected leader, sending a full tablet report...
I20260927 12:46:36.527577 7480 server_base.cc:1034] running on GCE node
W20260927 12:46:36.527485 7488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:36.527527 7485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:36.527484 7486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:36.528208 7480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:46:36.528882 7480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:46:36.530130 7480 hybrid_clock.cc:648] HybridClock initialized: now 1790513196530104 us; error 36 us; skew 500 ppm
Sep 27 12:46:36 dist-test-slave-lkn4 krb5kdc[6464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513196, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:46:36.532990 7480 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 12:46:36.533939 7480 webserver.cc:460] Webserver started at http://127.0.98.2:40097/ using document root <none> and password file <none>
I20260927 12:46:36.534539 7480 fs_manager.cc:359] Metadata directory not provided
I20260927 12:46:36.534628 7480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:46:36.539024 7480 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 12:46:36.541376 7495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:36.542882 7480 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 12:46:36.543020 7480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-1/wal
uuid: "5752b93fbf53475cb5cf0593fd908fed"
format_stamp: "Formatted at 2026-09-27 12:46:35 on dist-test-slave-lkn4"
server_key: "d7b9e5dabc505f2e79af83161095d7e6"
server_key_iv: "b899991ee2f1f1ed3560523b7515d7bd"
server_key_version: "encryptionkey@0"
I20260927 12:46:36.543586 7480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:46:36.558741 7480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:46:36.561924 7480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:46:36.562139 7480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:46:36.562783 7480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:46:36.563648 7480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:46:36.563696 7480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:36.563759 7480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:46:36.563786 7480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:36.572638 7480 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41427
I20260927 12:46:36.572674 7608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41427 every 8 connection(s)
I20260927 12:46:36.573763 7480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:46:36.579241 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 7480
I20260927 12:46:36.579430 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 6839
I20260927 12:46:36.585896 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:45825
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40735
--webserver_port=40971
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:40735,127.0.98.61:41239
--tserver_master_addrs=127.0.98.62:40735,127.0.98.61:41239,127.0.98.60:45213 with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 12:46:36 dist-test-slave-lkn4 krb5kdc[6464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513196, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 27 12:46:36 dist-test-slave-lkn4 krb5kdc[6464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513196, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 27 12:46:36 dist-test-slave-lkn4 krb5kdc[6464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513196, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 12:46:36.591828 7232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:36.577460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41285 (local address 127.0.98.61:41239)
0927 12:46:36.577573 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:36.577577 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:36.577670 (+ 93us) server_negotiation.cc:407] Connection header received
0927 12:46:36.578225 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:36.578228 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:36.578282 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:36.578330 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:36.581125 (+ 2795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.581931 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.583982 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.584330 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.584404 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:36.588145 (+ 3741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:36.588149 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:36.588152 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:36.588182 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:36.589905 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:36.590362 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:36.590365 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:36.590367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:36.590439 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:36.590743 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:36.590746 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:36.590748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:36.590965 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:36.591036 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:36.591536 (+ 500us) server_negotiation.cc:299] Negotiation successful
0927 12:46:36.591672 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 12:46:36.591828 6548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:36.575639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37415 (local address 127.0.98.62:40735)
0927 12:46:36.575795 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:36.575801 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:36.576915 (+ 1114us) server_negotiation.cc:407] Connection header received
0927 12:46:36.578233 (+ 1318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:36.578237 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:36.578295 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:36.578348 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:36.581139 (+ 2791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.581943 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.583983 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.584335 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.584412 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:36.587564 (+ 3152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:36.587568 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:36.587571 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:36.587603 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:36.589469 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:36.590065 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:36.590069 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:36.590070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:36.590153 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:36.590537 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:36.590540 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:36.590542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:36.590762 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:36.590838 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:36.591538 (+ 700us) server_negotiation.cc:299] Negotiation successful
0927 12:46:36.591658 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 12:46:36.592271 7476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:36.575712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57373 (local address 127.0.98.60:45213)
0927 12:46:36.575842 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:36.575847 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:36.577141 (+ 1294us) server_negotiation.cc:407] Connection header received
0927 12:46:36.578211 (+ 1070us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:36.578215 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:36.578273 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:36.578348 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:36.581132 (+ 2784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.581950 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.583995 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.584353 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.584419 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:36.588791 (+ 4372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:36.588795 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:36.588797 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:36.588832 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:36.590399 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:36.590896 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:36.590900 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:36.590902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:36.591000 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:36.591318 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:36.591322 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:36.591323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:36.591536 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:36.591615 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:36.591951 (+ 336us) server_negotiation.cc:299] Negotiation successful
0927 12:46:36.592083 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 12:46:36.593627 7615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:36.577359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41239 (local address 127.0.98.2:41285)
0927 12:46:36.577600 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:36.577611 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:36.577967 (+ 356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:36.578532 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:36.578539 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:36.579130 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:36.580946 (+ 1816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:36.580968 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.582130 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.582133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:36.583860 (+ 1727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:36.583873 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.584619 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.584622 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:36.585232 (+ 610us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:36.586388 (+ 1156us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:36.586403 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:36.587989 (+ 1586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:36.590018 (+ 2029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:36.590022 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:36.590023 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:36.590261 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:36.590536 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:36.590539 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:36.590541 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:36.590648 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:36.591084 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:36.591086 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:36.591256 (+ 170us) client_negotiation.cc:769] Sending connection context
0927 12:46:36.592097 (+ 841us) client_negotiation.cc:240] Negotiation successful
0927 12:46:36.592478 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":769,"spinlock_wait_cycles":10624,"thread_start_us":79,"threads_started":1}
I20260927 12:46:36.593631 7614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:36.576137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45213 (local address 127.0.98.2:57373)
0927 12:46:36.577054 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:36.577067 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:36.577963 (+ 896us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:36.578533 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:36.578541 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:36.579123 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:36.580953 (+ 1830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:36.580968 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.582130 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.582133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:36.583881 (+ 1748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:36.583889 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.584619 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.584623 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:36.585190 (+ 567us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:36.585949 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:36.585963 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:36.588656 (+ 2693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:36.590522 (+ 1866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:36.590525 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:36.590527 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:36.590765 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:36.591112 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:36.591115 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:36.591117 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:36.591218 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:36.591674 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:36.591677 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:36.591768 (+ 91us) client_negotiation.cc:769] Sending connection context
0927 12:46:36.592505 (+ 737us) client_negotiation.cc:240] Negotiation successful
0927 12:46:36.592671 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":829,"mutex_wait_us":882,"spinlock_wait_cycles":64256,"thread_start_us":148,"threads_started":1}
I20260927 12:46:36.593627 7613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:36.576137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40735 (local address 127.0.98.2:37415)
0927 12:46:36.576721 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:36.576804 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:36.577963 (+ 1159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:36.578533 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:36.578541 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:36.579123 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:36.580950 (+ 1827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:36.580968 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.582137 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.582139 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:36.583839 (+ 1700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:36.583851 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.584619 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.584623 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:36.584843 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:36.585477 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:36.585500 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:36.587356 (+ 1856us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:36.589599 (+ 2243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:36.589606 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:36.589623 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:36.589937 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:36.590260 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:36.590263 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:36.590265 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:36.590439 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:36.590899 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:36.590906 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:36.591256 (+ 350us) client_negotiation.cc:769] Sending connection context
0927 12:46:36.592529 (+ 1273us) client_negotiation.cc:240] Negotiation successful
0927 12:46:36.592726 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"mutex_wait_us":399,"spinlock_wait_cycles":29952,"thread_start_us":629,"threads_started":1}
I20260927 12:46:36.595065 7610 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40735
I20260927 12:46:36.595098 7609 heartbeater.cc:347] Connected to a master server at 127.0.98.60:45213
I20260927 12:46:36.595096 7611 heartbeater.cc:347] Connected to a master server at 127.0.98.61:41239
I20260927 12:46:36.595309 7609 heartbeater.cc:464] Registering TS with master...
I20260927 12:46:36.595309 7610 heartbeater.cc:464] Registering TS with master...
I20260927 12:46:36.595309 7611 heartbeater.cc:464] Registering TS with master...
I20260927 12:46:36.595897 7611 heartbeater.cc:511] Master 127.0.98.61:41239 requested a full tablet report, sending...
I20260927 12:46:36.595909 7610 heartbeater.cc:511] Master 127.0.98.62:40735 requested a full tablet report, sending...
I20260927 12:46:36.595897 7609 heartbeater.cc:511] Master 127.0.98.60:45213 requested a full tablet report, sending...
I20260927 12:46:36.597009 6509 ts_manager.cc:194] Re-registered known tserver with Master: 5752b93fbf53475cb5cf0593fd908fed (127.0.98.2:41427)
I20260927 12:46:36.597050 7285 ts_manager.cc:194] Registered new tserver with Master: 5752b93fbf53475cb5cf0593fd908fed (127.0.98.2:41427)
I20260927 12:46:36.597486 7109 ts_manager.cc:194] Registered new tserver with Master: 5752b93fbf53475cb5cf0593fd908fed (127.0.98.2:41427)
I20260927 12:46:36.597786 6509 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37415
I20260927 12:46:36.699044 7265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:36.695046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45213 (local address 127.0.0.1:40302)
0927 12:46:36.695357 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:36.695372 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:36.695490 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:36.696034 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:36.696038 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:36.696066 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:46:36.696203 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:36.696211 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.697072 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.697076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:36.697877 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:36.697888 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.698603 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.698607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:36.698700 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:36.698724 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 12:46:36.698801 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 12:46:36.698891 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219}
I20260927 12:46:36.699034 7476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:36.695603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40302 (local address 127.0.98.60:45213)
0927 12:46:36.695756 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:36.695761 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:36.695776 (+ 15us) server_negotiation.cc:407] Connection header received
0927 12:46:36.695827 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:36.695830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:36.695890 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:36.695990 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:46:36.696346 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.696948 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.698005 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.698479 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.698578 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:36.698747 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:36.698824 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 12:46:36.698899 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 12:46:36.699859 7294 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260927 12:46:36.700269 7332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 205c8e3f55144799958e82a465ac9b66. Latest consensus state: current_term: 1 leader_uuid: "205c8e3f55144799958e82a465ac9b66" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "205c8e3f55144799958e82a465ac9b66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40735 } } peers { permanent_uuid: "4fdbce7f7c3d4fda8c562a4fb4072671" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41239 } attrs { promote: false } } peers { permanent_uuid: "70b88e0990ca471d93129ebffc98d9cf" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45213 } attrs { promote: true } } }
I20260927 12:46:36.700412 7332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf [sys.catalog]: This master's current role is: LEARNER
W20260927 12:46:36.704916 7616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:46:36.705188 7616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:46:36.705246 7616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:46:36.709470 7616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:46:36.709546 7616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:46:36.709635 7616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:46:36.715216 7616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:45825
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=40971
--tserver_master_addrs=127.0.98.62:40735,127.0.98.61:41239,127.0.98.60:45213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:46:36.716298 7616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:46:36.717363 7616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:46:36.727099 7621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:36.727125 7622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:36.727106 7624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:36.727339 7616 server_base.cc:1034] running on GCE node
I20260927 12:46:36.727874 7616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:46:36.728452 7616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:46:36.729660 7616 hybrid_clock.cc:648] HybridClock initialized: now 1790513196729622 us; error 35 us; skew 500 ppm
Sep 27 12:46:36 dist-test-slave-lkn4 krb5kdc[6464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513196, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:46:36.732498 7616 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 12:46:36.733490 7616 webserver.cc:460] Webserver started at http://127.0.98.3:40971/ using document root <none> and password file <none>
I20260927 12:46:36.734345 7616 fs_manager.cc:359] Metadata directory not provided
I20260927 12:46:36.734444 7616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:46:36.738817 7616 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 12:46:36.741039 7631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:36.742501 7616 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:46:36.742620 7616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-2/wal
uuid: "ae19c5cb5e6d4abc8d5bf1463c817642"
format_stamp: "Formatted at 2026-09-27 12:46:35 on dist-test-slave-lkn4"
server_key: "d73887f83eb19c4cda2060ed11a28650"
server_key_iv: "97f4c863c38b37e71eab4934e33b58a3"
server_key_version: "encryptionkey@0"
I20260927 12:46:36.743228 7616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:46:36.752300 7616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:46:36.755389 7616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:46:36.755573 7616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:46:36.756162 7616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:46:36.757011 7616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:46:36.757056 7616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:36.757110 7616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:46:36.757138 7616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:36.765516 7616 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45825
I20260927 12:46:36.765760 7744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45825 every 8 connection(s)
I20260927 12:46:36.766670 7616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:46:36.774950 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 7616
Sep 27 12:46:36 dist-test-slave-lkn4 krb5kdc[6464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513196, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 27 12:46:36 dist-test-slave-lkn4 krb5kdc[6464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513196, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 27 12:46:36 dist-test-slave-lkn4 krb5kdc[6464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513196, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 12:46:36.784564 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:36.768983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41239 (local address 127.0.98.3:34561)
0927 12:46:36.769884 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:36.769897 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:36.770668 (+ 771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:36.771190 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:36.771193 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:36.771706 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:36.772305 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:36.772316 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.773150 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.773152 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:36.774172 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:36.774181 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.774907 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.774911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:36.775148 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:36.775821 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:36.775842 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:36.779322 (+ 3480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:36.781313 (+ 1991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:36.781317 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:36.781319 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:36.782259 (+ 940us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:36.782553 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:36.782556 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:36.782558 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:36.782633 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:36.783272 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:36.783275 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:36.783366 (+ 91us) client_negotiation.cc:769] Sending connection context
0927 12:46:36.783449 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 12:46:36.783580 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":816,"mutex_wait_us":217,"spinlock_wait_cycles":87168,"thread_start_us":146,"threads_started":1}
I20260927 12:46:36.784850 7751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:36.770178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45213 (local address 127.0.98.3:37139)
0927 12:46:36.770405 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:36.770417 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:36.770671 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:36.771165 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:36.771172 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:36.771722 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:36.772300 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:36.772314 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.773140 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.773143 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:36.774243 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:36.774251 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.774907 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.774911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:36.775189 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:36.776068 (+ 879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:36.776082 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:36.778446 (+ 2364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:36.781066 (+ 2620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:36.781073 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:36.781090 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:36.781443 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:36.781813 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:36.781816 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:36.781818 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:36.781993 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:36.782469 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:36.782476 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:36.782815 (+ 339us) client_negotiation.cc:769] Sending connection context
0927 12:46:36.783047 (+ 232us) client_negotiation.cc:240] Negotiation successful
0927 12:46:36.783407 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":212,"spinlock_wait_cycles":1408,"thread_start_us":68,"threads_started":1}
I20260927 12:46:36.785343 7476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:36.770300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37139 (local address 127.0.98.60:45213)
0927 12:46:36.770424 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:36.770429 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:36.770469 (+ 40us) server_negotiation.cc:407] Connection header received
0927 12:46:36.770865 (+ 396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:36.770868 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:36.770924 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:36.770991 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:36.772509 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.773006 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.774331 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.774647 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.774742 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:36.778639 (+ 3897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:36.778643 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:36.778646 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:36.778673 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:36.780934 (+ 2261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:36.781553 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:36.781557 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:36.781559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:36.781643 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:36.782127 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:36.782130 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:36.782132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:36.782341 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:36.782421 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:36.785068 (+ 2647us) server_negotiation.cc:299] Negotiation successful
0927 12:46:36.785187 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 12:46:36.786428 7745 heartbeater.cc:347] Connected to a master server at 127.0.98.60:45213
I20260927 12:46:36.786684 7745 heartbeater.cc:464] Registering TS with master...
I20260927 12:46:36.787307 7745 heartbeater.cc:511] Master 127.0.98.60:45213 requested a full tablet report, sending...
I20260927 12:46:36.788604 7285 ts_manager.cc:194] Registered new tserver with Master: ae19c5cb5e6d4abc8d5bf1463c817642 (127.0.98.3:45825)
I20260927 12:46:36.788895 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:36.778424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59146 (local address 127.0.98.62:40735)
0927 12:46:36.779909 (+ 1485us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:36.779921 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:36.779933 (+ 12us) server_negotiation.cc:407] Connection header received
0927 12:46:36.779971 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:36.779973 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:36.780007 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:36.780084 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:36.780967 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.781746 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.782874 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.783196 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.783328 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:36.784904 (+ 1576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:36.784908 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:36.784910 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:36.784938 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:36.786774 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:36.787190 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:36.787193 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:36.787195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:36.787278 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:36.787559 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:36.787563 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:36.787564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:36.787801 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:36.787959 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:36.788570 (+ 611us) server_negotiation.cc:299] Negotiation successful
0927 12:46:36.788695 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1411,"thread_start_us":82,"threads_started":1}
I20260927 12:46:36.789470 7749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:36.768983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40735 (local address 127.0.98.3:52379)
0927 12:46:36.769571 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:36.769617 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:36.770668 (+ 1051us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:36.771166 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:36.771173 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:36.771714 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:36.772301 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:36.772316 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.774161 (+ 1845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.774164 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:36.774760 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:36.774770 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.775451 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.775455 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:36.775540 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:36.775957 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:36.775972 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:36.778792 (+ 2820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:36.785359 (+ 6567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:36.785362 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:36.785364 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:36.785601 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:36.786234 (+ 633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:36.786237 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:36.786238 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:36.786322 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:36.788762 (+ 2440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:36.788765 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:36.788868 (+ 103us) client_negotiation.cc:769] Sending connection context
0927 12:46:36.788974 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 12:46:36.789120 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":82,"spinlock_wait_cycles":82048,"thread_start_us":639,"threads_started":1}
I20260927 12:46:36.789819 7232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:36.768453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34561 (local address 127.0.98.61:41239)
0927 12:46:36.768580 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:36.768585 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:36.769971 (+ 1386us) server_negotiation.cc:407] Connection header received
0927 12:46:36.770900 (+ 929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:36.770904 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:36.770958 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:36.771008 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:36.772481 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.773049 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.774290 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.774625 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.774749 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:36.779494 (+ 4745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:36.779498 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:36.779501 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:36.779528 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:36.781188 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:36.782381 (+ 1193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:36.782384 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:36.782386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:36.782466 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:36.782717 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:36.782720 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:36.782722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:36.782915 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:36.782990 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:36.788285 (+ 5295us) server_negotiation.cc:299] Negotiation successful
0927 12:46:36.789675 (+ 1390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1287,"server-negotiator.queue_time_us":40}
I20260927 12:46:36.790340 7761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:36.778643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48112 (local address 127.0.98.61:41239)
0927 12:46:36.778913 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:36.778917 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:36.778931 (+ 14us) server_negotiation.cc:407] Connection header received
0927 12:46:36.778968 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:36.778971 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:36.779017 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:36.779101 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:36.780263 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.781044 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.782185 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.782517 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.782602 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:36.784244 (+ 1642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:36.784247 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:36.784250 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:36.784274 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:36.786371 (+ 2097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:36.786745 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:36.786748 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:36.786750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:36.786826 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:36.787092 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:36.787095 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:36.787097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:36.787309 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:36.787418 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:36.788059 (+ 641us) server_negotiation.cc:299] Negotiation successful
0927 12:46:36.790124 (+ 2065us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":80,"threads_started":1}
I20260927 12:46:36.790441 7747 heartbeater.cc:347] Connected to a master server at 127.0.98.61:41239
I20260927 12:46:36.790529 7747 heartbeater.cc:464] Registering TS with master...
I20260927 12:46:36.790715 6548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:36.768612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52379 (local address 127.0.98.62:40735)
0927 12:46:36.768711 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:36.768715 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:36.769743 (+ 1028us) server_negotiation.cc:407] Connection header received
0927 12:46:36.770901 (+ 1158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:36.770904 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:36.770958 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:36.771008 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:36.773200 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.773998 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.774907 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.775203 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.775274 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:36.778947 (+ 3673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:36.778951 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:36.778954 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:36.778981 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:36.781996 (+ 3015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:36.785992 (+ 3996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:36.785996 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:36.785997 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:36.786107 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:36.788367 (+ 2260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:36.788370 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:36.788372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:36.788575 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:36.788650 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:36.790427 (+ 1777us) server_negotiation.cc:299] Negotiation successful
0927 12:46:36.790560 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29,"spinlock_wait_cycles":3968}
I20260927 12:46:36.790805 7747 heartbeater.cc:511] Master 127.0.98.61:41239 requested a full tablet report, sending...
I20260927 12:46:36.791452 7746 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40735
I20260927 12:46:36.791541 7746 heartbeater.cc:464] Registering TS with master...
I20260927 12:46:36.791736 7746 heartbeater.cc:511] Master 127.0.98.62:40735 requested a full tablet report, sending...
I20260927 12:46:36.791749 7110 ts_manager.cc:194] Registered new tserver with Master: ae19c5cb5e6d4abc8d5bf1463c817642 (127.0.98.3:45825)
I20260927 12:46:36.792316 6507 ts_manager.cc:194] Re-registered known tserver with Master: ae19c5cb5e6d4abc8d5bf1463c817642 (127.0.98.3:45825)
I20260927 12:46:36.793040 6507 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52379
I20260927 12:46:36.792379 6509 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59146:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:46:36.795742 6509 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:46:36.797470 7759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:36.778247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40308 (local address 127.0.98.60:45213)
0927 12:46:36.783849 (+ 5602us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:36.783861 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:36.786869 (+ 3008us) server_negotiation.cc:407] Connection header received
0927 12:46:36.786911 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:36.786914 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:36.786959 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:36.787792 (+ 833us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:36.787927 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.788631 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.789875 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.792390 (+ 2515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.792547 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:36.794130 (+ 1583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:36.794135 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:36.794138 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:36.794167 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:36.795781 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:36.796217 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:36.796221 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:36.796224 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:36.796312 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:36.796589 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:36.796592 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:36.796594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:36.796788 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:36.796910 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:36.797159 (+ 249us) server_negotiation.cc:299] Negotiation successful
0927 12:46:36.797287 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5522,"thread_start_us":78,"threads_started":1}
I20260927 12:46:36.798463 7068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 205c8e3f55144799958e82a465ac9b66 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "205c8e3f55144799958e82a465ac9b66" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "205c8e3f55144799958e82a465ac9b66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40735 } } peers { permanent_uuid: "4fdbce7f7c3d4fda8c562a4fb4072671" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41239 } attrs { promote: false } } peers { permanent_uuid: "70b88e0990ca471d93129ebffc98d9cf" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 45213 } attrs { promote: true } } }
I20260927 12:46:36.798602 7068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 205c8e3f55144799958e82a465ac9b66 [sys.catalog]: This master's current role is: LEADER
I20260927 12:46:36.801610 7764 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 205c8e3f55144799958e82a465ac9b66: attempting to promote NON_VOTER 70b88e0990ca471d93129ebffc98d9cf to VOTER
I20260927 12:46:36.801924 7764 consensus_queue.cc:237] T 00000000000000000000000000000000 P 205c8e3f55144799958e82a465ac9b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "205c8e3f55144799958e82a465ac9b66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40735 } } peers { permanent_uuid: "4fdbce7f7c3d4fda8c562a4fb4072671" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41239 } attrs { promote: false } } peers { permanent_uuid: "70b88e0990ca471d93129ebffc98d9cf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45213 } attrs { promote: false } }
I20260927 12:46:36.803156 7294 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf [term 1 LEARNER]: Refusing update from remote peer 205c8e3f55144799958e82a465ac9b66: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260927 12:46:36.803264 7119 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4fdbce7f7c3d4fda8c562a4fb4072671 [term 1 FOLLOWER]: Refusing update from remote peer 205c8e3f55144799958e82a465ac9b66: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260927 12:46:36.803578 7764 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 205c8e3f55144799958e82a465ac9b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70b88e0990ca471d93129ebffc98d9cf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45213 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260927 12:46:36.803771 7068 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 205c8e3f55144799958e82a465ac9b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fdbce7f7c3d4fda8c562a4fb4072671" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41239 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260927 12:46:36.805774 7763 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 205c8e3f55144799958e82a465ac9b66 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 70b88e0990ca471d93129ebffc98d9cf (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "205c8e3f55144799958e82a465ac9b66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40735 } } peers { permanent_uuid: "4fdbce7f7c3d4fda8c562a4fb4072671" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41239 } attrs { promote: false } } peers { permanent_uuid: "70b88e0990ca471d93129ebffc98d9cf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45213 } attrs { promote: false } } }
I20260927 12:46:36.806213 7119 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4fdbce7f7c3d4fda8c562a4fb4072671 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 70b88e0990ca471d93129ebffc98d9cf (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "205c8e3f55144799958e82a465ac9b66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40735 } } peers { permanent_uuid: "4fdbce7f7c3d4fda8c562a4fb4072671" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41239 } attrs { promote: false } } peers { permanent_uuid: "70b88e0990ca471d93129ebffc98d9cf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45213 } attrs { promote: false } } }
I20260927 12:46:36.807361 7764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 205c8e3f55144799958e82a465ac9b66 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "205c8e3f55144799958e82a465ac9b66" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "205c8e3f55144799958e82a465ac9b66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40735 } } peers { permanent_uuid: "4fdbce7f7c3d4fda8c562a4fb4072671" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41239 } attrs { promote: false } } peers { permanent_uuid: "70b88e0990ca471d93129ebffc98d9cf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45213 } attrs { promote: false } } }
I20260927 12:46:36.807405 7763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 205c8e3f55144799958e82a465ac9b66 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "205c8e3f55144799958e82a465ac9b66" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "205c8e3f55144799958e82a465ac9b66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40735 } } peers { permanent_uuid: "4fdbce7f7c3d4fda8c562a4fb4072671" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41239 } attrs { promote: false } } peers { permanent_uuid: "70b88e0990ca471d93129ebffc98d9cf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45213 } attrs { promote: false } } }
I20260927 12:46:36.807497 7764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 205c8e3f55144799958e82a465ac9b66 [sys.catalog]: This master's current role is: LEADER
I20260927 12:46:36.807186 7294 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 70b88e0990ca471d93129ebffc98d9cf (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "205c8e3f55144799958e82a465ac9b66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40735 } } peers { permanent_uuid: "4fdbce7f7c3d4fda8c562a4fb4072671" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41239 } attrs { promote: false } } peers { permanent_uuid: "70b88e0990ca471d93129ebffc98d9cf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45213 } attrs { promote: false } } }
I20260927 12:46:36.807533 7763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 205c8e3f55144799958e82a465ac9b66 [sys.catalog]: This master's current role is: LEADER
I20260927 12:46:36.807673 7764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 205c8e3f55144799958e82a465ac9b66 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "205c8e3f55144799958e82a465ac9b66" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "205c8e3f55144799958e82a465ac9b66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40735 } } peers { permanent_uuid: "4fdbce7f7c3d4fda8c562a4fb4072671" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41239 } attrs { promote: false } } peers { permanent_uuid: "70b88e0990ca471d93129ebffc98d9cf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45213 } attrs { promote: false } } }
I20260927 12:46:36.807830 7764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 205c8e3f55144799958e82a465ac9b66 [sys.catalog]: This master's current role is: LEADER
I20260927 12:46:36.808374 7772 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fdbce7f7c3d4fda8c562a4fb4072671 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "205c8e3f55144799958e82a465ac9b66" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "205c8e3f55144799958e82a465ac9b66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40735 } } peers { permanent_uuid: "4fdbce7f7c3d4fda8c562a4fb4072671" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41239 } attrs { promote: false } } peers { permanent_uuid: "70b88e0990ca471d93129ebffc98d9cf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45213 } attrs { promote: false } } }
I20260927 12:46:36.808519 7772 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fdbce7f7c3d4fda8c562a4fb4072671 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:46:36.809036 7332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "205c8e3f55144799958e82a465ac9b66" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "205c8e3f55144799958e82a465ac9b66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40735 } } peers { permanent_uuid: "4fdbce7f7c3d4fda8c562a4fb4072671" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41239 } attrs { promote: false } } peers { permanent_uuid: "70b88e0990ca471d93129ebffc98d9cf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45213 } attrs { promote: false } } }
I20260927 12:46:36.809137 7332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:46:36.818444 7265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:36.812516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43975 (local address 127.0.0.1:39298)
0927 12:46:36.812642 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:36.812657 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:36.812738 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:36.814025 (+ 1287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:36.814029 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:36.814072 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:46:36.814178 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:36.814184 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.815845 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.815848 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:36.816664 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:36.816671 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.817990 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.817993 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:36.818138 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:36.818161 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 12:46:36.818226 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 12:46:36.818309 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 12:46:36.819062 7773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:36.812675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39298 (local address 127.0.98.1:43975)
0927 12:46:36.812980 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:36.812992 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:36.813014 (+ 22us) server_negotiation.cc:407] Connection header received
0927 12:46:36.813095 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:36.813100 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:36.813864 (+ 764us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:36.814364 (+ 500us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:46:36.814495 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.815730 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.816780 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.817637 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.817744 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:36.817926 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:36.818813 (+ 887us) server_negotiation.cc:299] Negotiation successful
0927 12:46:36.818914 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":107,"threads_started":1}
I20260927 12:46:36.821401 7777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:36.813318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45825 (local address 127.0.0.1:45008)
0927 12:46:36.814618 (+ 1300us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:36.814629 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:36.814697 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:36.816532 (+ 1835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:36.816535 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:36.816553 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:46:36.816640 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:36.816645 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.818853 (+ 2208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.818856 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:36.820018 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:36.820029 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.821028 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.821032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:36.821113 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:36.821134 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 12:46:36.821195 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 12:46:36.821265 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1243,"mutex_wait_us":26,"thread_start_us":393,"threads_started":1}
I20260927 12:46:36.821859 7776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:36.813465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45008 (local address 127.0.98.3:45825)
0927 12:46:36.816042 (+ 2577us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:36.816053 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:36.816077 (+ 24us) server_negotiation.cc:407] Connection header received
0927 12:46:36.816156 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:36.816162 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:36.816392 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:36.816540 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:46:36.816787 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.818710 (+ 1923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.820165 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.820728 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.820834 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:36.820978 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:36.821653 (+ 675us) server_negotiation.cc:299] Negotiation successful
0927 12:46:36.821747 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2457,"thread_start_us":98,"threads_started":1}
I20260927 12:46:36.822325 7406 tablet_service.cc:1467] Processing CreateTablet for tablet efc617f1ea154360b636e3dddcd1fed1 (DEFAULT_TABLE table=test-table [id=4db12233c27148f59199c2fa4d0dcff5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:46:36.823444 7406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efc617f1ea154360b636e3dddcd1fed1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:46:36.828511 7679 tablet_service.cc:1467] Processing CreateTablet for tablet efc617f1ea154360b636e3dddcd1fed1 (DEFAULT_TABLE table=test-table [id=4db12233c27148f59199c2fa4d0dcff5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:46:36.829519 7679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efc617f1ea154360b636e3dddcd1fed1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:46:36.830559 7774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:36.812983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41427 (local address 127.0.0.1:42992)
0927 12:46:36.813219 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:36.813232 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:36.814437 (+ 1205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:36.817264 (+ 2827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:36.817267 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:36.817282 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:46:36.817374 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:36.817379 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.819032 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.819035 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:36.827225 (+ 8190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:36.827236 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.830128 (+ 2892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.830132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:36.830233 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:36.830260 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 12:46:36.830332 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 12:46:36.830417 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":26,"thread_start_us":105,"threads_started":1}
I20260927 12:46:36.830585 7775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:36.813143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42992 (local address 127.0.98.2:41427)
0927 12:46:36.816826 (+ 3683us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:36.816837 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:36.816858 (+ 21us) server_negotiation.cc:407] Connection header received
0927 12:46:36.816926 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:36.816931 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:36.817109 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:36.817501 (+ 392us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:46:36.817626 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.818923 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.829111 (+ 10188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.829946 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.830136 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:36.830297 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:36.830357 (+ 60us) server_negotiation.cc:299] Negotiation successful
0927 12:46:36.830447 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3577,"thread_start_us":144,"threads_started":1}
I20260927 12:46:36.832934 7778 tablet_bootstrap.cc:492] T efc617f1ea154360b636e3dddcd1fed1 P 9387b2fa57714be0aa7b8e69be342b8e: Bootstrap starting.
I20260927 12:46:36.833810 7543 tablet_service.cc:1467] Processing CreateTablet for tablet efc617f1ea154360b636e3dddcd1fed1 (DEFAULT_TABLE table=test-table [id=4db12233c27148f59199c2fa4d0dcff5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:46:36.834798 7543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efc617f1ea154360b636e3dddcd1fed1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:46:36.835464 7778 tablet_bootstrap.cc:654] T efc617f1ea154360b636e3dddcd1fed1 P 9387b2fa57714be0aa7b8e69be342b8e: Neither blocks nor log segments found. Creating new log.
I20260927 12:46:36.836352 7778 log.cc:824] T efc617f1ea154360b636e3dddcd1fed1 P 9387b2fa57714be0aa7b8e69be342b8e: Log is configured to *not* fsync() on all Append() calls
I20260927 12:46:36.838495 7780 tablet_bootstrap.cc:492] T efc617f1ea154360b636e3dddcd1fed1 P ae19c5cb5e6d4abc8d5bf1463c817642: Bootstrap starting.
I20260927 12:46:36.840490 7778 tablet_bootstrap.cc:492] T efc617f1ea154360b636e3dddcd1fed1 P 9387b2fa57714be0aa7b8e69be342b8e: No bootstrap required, opened a new log
I20260927 12:46:36.840703 7778 ts_tablet_manager.cc:1397] T efc617f1ea154360b636e3dddcd1fed1 P 9387b2fa57714be0aa7b8e69be342b8e: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260927 12:46:36.841305 7780 tablet_bootstrap.cc:654] T efc617f1ea154360b636e3dddcd1fed1 P ae19c5cb5e6d4abc8d5bf1463c817642: Neither blocks nor log segments found. Creating new log.
I20260927 12:46:36.842226 7781 tablet_bootstrap.cc:492] T efc617f1ea154360b636e3dddcd1fed1 P 5752b93fbf53475cb5cf0593fd908fed: Bootstrap starting.
I20260927 12:46:36.842412 7780 log.cc:824] T efc617f1ea154360b636e3dddcd1fed1 P ae19c5cb5e6d4abc8d5bf1463c817642: Log is configured to *not* fsync() on all Append() calls
I20260927 12:46:36.844199 7780 tablet_bootstrap.cc:492] T efc617f1ea154360b636e3dddcd1fed1 P ae19c5cb5e6d4abc8d5bf1463c817642: No bootstrap required, opened a new log
I20260927 12:46:36.844390 7780 ts_tablet_manager.cc:1397] T efc617f1ea154360b636e3dddcd1fed1 P ae19c5cb5e6d4abc8d5bf1463c817642: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 12:46:36.844714 7781 tablet_bootstrap.cc:654] T efc617f1ea154360b636e3dddcd1fed1 P 5752b93fbf53475cb5cf0593fd908fed: Neither blocks nor log segments found. Creating new log.
I20260927 12:46:36.844954 7778 raft_consensus.cc:348] T efc617f1ea154360b636e3dddcd1fed1 P 9387b2fa57714be0aa7b8e69be342b8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387b2fa57714be0aa7b8e69be342b8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43975 } } peers { permanent_uuid: "5752b93fbf53475cb5cf0593fd908fed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41427 } } peers { permanent_uuid: "ae19c5cb5e6d4abc8d5bf1463c817642" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45825 } }
I20260927 12:46:36.845232 7778 raft_consensus.cc:374] T efc617f1ea154360b636e3dddcd1fed1 P 9387b2fa57714be0aa7b8e69be342b8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:46:36.845299 7778 raft_consensus.cc:729] T efc617f1ea154360b636e3dddcd1fed1 P 9387b2fa57714be0aa7b8e69be342b8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9387b2fa57714be0aa7b8e69be342b8e, State: Initialized, Role: FOLLOWER
I20260927 12:46:36.845553 7781 log.cc:824] T efc617f1ea154360b636e3dddcd1fed1 P 5752b93fbf53475cb5cf0593fd908fed: Log is configured to *not* fsync() on all Append() calls
I20260927 12:46:36.845824 7778 consensus_queue.cc:260] T efc617f1ea154360b636e3dddcd1fed1 P 9387b2fa57714be0aa7b8e69be342b8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387b2fa57714be0aa7b8e69be342b8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43975 } } peers { permanent_uuid: "5752b93fbf53475cb5cf0593fd908fed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41427 } } peers { permanent_uuid: "ae19c5cb5e6d4abc8d5bf1463c817642" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45825 } }
I20260927 12:46:36.846977 7778 ts_tablet_manager.cc:1428] T efc617f1ea154360b636e3dddcd1fed1 P 9387b2fa57714be0aa7b8e69be342b8e: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 12:46:36.847154 7473 heartbeater.cc:503] Master 127.0.98.62:40735 was elected leader, sending a full tablet report...
I20260927 12:46:36.847985 7781 tablet_bootstrap.cc:492] T efc617f1ea154360b636e3dddcd1fed1 P 5752b93fbf53475cb5cf0593fd908fed: No bootstrap required, opened a new log
I20260927 12:46:36.848176 7781 ts_tablet_manager.cc:1397] T efc617f1ea154360b636e3dddcd1fed1 P 5752b93fbf53475cb5cf0593fd908fed: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 12:46:36.849231 7780 raft_consensus.cc:348] T efc617f1ea154360b636e3dddcd1fed1 P ae19c5cb5e6d4abc8d5bf1463c817642 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387b2fa57714be0aa7b8e69be342b8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43975 } } peers { permanent_uuid: "5752b93fbf53475cb5cf0593fd908fed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41427 } } peers { permanent_uuid: "ae19c5cb5e6d4abc8d5bf1463c817642" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45825 } }
I20260927 12:46:36.849483 7780 raft_consensus.cc:374] T efc617f1ea154360b636e3dddcd1fed1 P ae19c5cb5e6d4abc8d5bf1463c817642 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:46:36.849525 7780 raft_consensus.cc:729] T efc617f1ea154360b636e3dddcd1fed1 P ae19c5cb5e6d4abc8d5bf1463c817642 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae19c5cb5e6d4abc8d5bf1463c817642, State: Initialized, Role: FOLLOWER
I20260927 12:46:36.850025 7780 consensus_queue.cc:260] T efc617f1ea154360b636e3dddcd1fed1 P ae19c5cb5e6d4abc8d5bf1463c817642 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387b2fa57714be0aa7b8e69be342b8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43975 } } peers { permanent_uuid: "5752b93fbf53475cb5cf0593fd908fed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41427 } } peers { permanent_uuid: "ae19c5cb5e6d4abc8d5bf1463c817642" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45825 } }
I20260927 12:46:36.851164 7780 ts_tablet_manager.cc:1428] T efc617f1ea154360b636e3dddcd1fed1 P ae19c5cb5e6d4abc8d5bf1463c817642: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20260927 12:46:36.852228 7746 heartbeater.cc:503] Master 127.0.98.62:40735 was elected leader, sending a full tablet report...
I20260927 12:46:36.853500 7781 raft_consensus.cc:348] T efc617f1ea154360b636e3dddcd1fed1 P 5752b93fbf53475cb5cf0593fd908fed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387b2fa57714be0aa7b8e69be342b8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43975 } } peers { permanent_uuid: "5752b93fbf53475cb5cf0593fd908fed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41427 } } peers { permanent_uuid: "ae19c5cb5e6d4abc8d5bf1463c817642" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45825 } }
I20260927 12:46:36.853847 7781 raft_consensus.cc:374] T efc617f1ea154360b636e3dddcd1fed1 P 5752b93fbf53475cb5cf0593fd908fed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:46:36.853945 7781 raft_consensus.cc:729] T efc617f1ea154360b636e3dddcd1fed1 P 5752b93fbf53475cb5cf0593fd908fed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5752b93fbf53475cb5cf0593fd908fed, State: Initialized, Role: FOLLOWER
I20260927 12:46:36.854522 7781 consensus_queue.cc:260] T efc617f1ea154360b636e3dddcd1fed1 P 5752b93fbf53475cb5cf0593fd908fed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387b2fa57714be0aa7b8e69be342b8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43975 } } peers { permanent_uuid: "5752b93fbf53475cb5cf0593fd908fed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41427 } } peers { permanent_uuid: "ae19c5cb5e6d4abc8d5bf1463c817642" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45825 } }
I20260927 12:46:36.855580 7781 ts_tablet_manager.cc:1428] T efc617f1ea154360b636e3dddcd1fed1 P 5752b93fbf53475cb5cf0593fd908fed: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260927 12:46:36.856702 7610 heartbeater.cc:503] Master 127.0.98.62:40735 was elected leader, sending a full tablet report...
W20260927 12:46:36.879396 7475 tablet.cc:2369] T efc617f1ea154360b636e3dddcd1fed1 P 9387b2fa57714be0aa7b8e69be342b8e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:46:36.897799 7786 raft_consensus.cc:482] T efc617f1ea154360b636e3dddcd1fed1 P 5752b93fbf53475cb5cf0593fd908fed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:46:36.898006 7786 raft_consensus.cc:504] T efc617f1ea154360b636e3dddcd1fed1 P 5752b93fbf53475cb5cf0593fd908fed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387b2fa57714be0aa7b8e69be342b8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43975 } } peers { permanent_uuid: "5752b93fbf53475cb5cf0593fd908fed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41427 } } peers { permanent_uuid: "ae19c5cb5e6d4abc8d5bf1463c817642" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45825 } }
I20260927 12:46:36.899402 7786 leader_election.cc:288] T efc617f1ea154360b636e3dddcd1fed1 P 5752b93fbf53475cb5cf0593fd908fed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9387b2fa57714be0aa7b8e69be342b8e (127.0.98.1:43975), ae19c5cb5e6d4abc8d5bf1463c817642 (127.0.98.3:45825)
I20260927 12:46:36.902998 7773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:36.899751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35165 (local address 127.0.98.1:43975)
0927 12:46:36.899887 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:36.899892 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:36.899911 (+ 19us) server_negotiation.cc:407] Connection header received
0927 12:46:36.900086 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:36.900088 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:36.900130 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:36.900207 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:46:36.900601 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.901084 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.902113 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.902485 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.902577 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:36.902634 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:36.902834 (+ 200us) server_negotiation.cc:299] Negotiation successful
0927 12:46:36.902898 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 12:46:36.903007 7776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:36.899896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53983 (local address 127.0.98.3:45825)
0927 12:46:36.900016 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:36.900020 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:36.900032 (+ 12us) server_negotiation.cc:407] Connection header received
0927 12:46:36.900082 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:36.900085 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:36.900129 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:36.900201 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:46:36.900474 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.901001 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.902026 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.902462 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.902545 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:36.902609 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:36.902812 (+ 203us) server_negotiation.cc:299] Negotiation successful
0927 12:46:36.902880 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 12:46:36.902998 7613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:36.899677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45825 (local address 127.0.98.2:53983)
0927 12:46:36.899855 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:36.899869 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:36.899975 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:36.900219 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:36.900222 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:36.900238 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:46:36.900346 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:36.900351 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.901099 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.901103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:36.901895 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:36.901903 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.902572 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.902576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:36.902706 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:36.902723 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 12:46:36.902778 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 12:46:36.902847 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":2,"spinlock_wait_cycles":9472}
I20260927 12:46:36.903008 7615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:36.899646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43975 (local address 127.0.98.2:35165)
0927 12:46:36.899851 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:36.899866 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:36.899974 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:36.900437 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:36.900439 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:36.900451 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:46:36.900511 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:36.900514 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.901191 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.901194 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:36.901970 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:36.901981 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.902593 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.902597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:36.902726 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:36.902742 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 12:46:36.902810 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 12:46:36.902879 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":32,"spinlock_wait_cycles":11904}
I20260927 12:46:36.903677 7699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc617f1ea154360b636e3dddcd1fed1" candidate_uuid: "5752b93fbf53475cb5cf0593fd908fed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae19c5cb5e6d4abc8d5bf1463c817642" is_pre_election: true
I20260927 12:46:36.903687 7426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc617f1ea154360b636e3dddcd1fed1" candidate_uuid: "5752b93fbf53475cb5cf0593fd908fed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9387b2fa57714be0aa7b8e69be342b8e" is_pre_election: true
I20260927 12:46:36.903963 7426 raft_consensus.cc:2454] T efc617f1ea154360b636e3dddcd1fed1 P 9387b2fa57714be0aa7b8e69be342b8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5752b93fbf53475cb5cf0593fd908fed in term 0.
I20260927 12:46:36.903963 7699 raft_consensus.cc:2454] T efc617f1ea154360b636e3dddcd1fed1 P ae19c5cb5e6d4abc8d5bf1463c817642 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5752b93fbf53475cb5cf0593fd908fed in term 0.
I20260927 12:46:36.904428 7499 leader_election.cc:302] T efc617f1ea154360b636e3dddcd1fed1 P 5752b93fbf53475cb5cf0593fd908fed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5752b93fbf53475cb5cf0593fd908fed, 9387b2fa57714be0aa7b8e69be342b8e; no voters:
I20260927 12:46:36.904755 7786 raft_consensus.cc:2781] T efc617f1ea154360b636e3dddcd1fed1 P 5752b93fbf53475cb5cf0593fd908fed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:46:36.904843 7786 raft_consensus.cc:482] T efc617f1ea154360b636e3dddcd1fed1 P 5752b93fbf53475cb5cf0593fd908fed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:46:36.904893 7786 raft_consensus.cc:3037] T efc617f1ea154360b636e3dddcd1fed1 P 5752b93fbf53475cb5cf0593fd908fed [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:46:36.905932 7786 raft_consensus.cc:504] T efc617f1ea154360b636e3dddcd1fed1 P 5752b93fbf53475cb5cf0593fd908fed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387b2fa57714be0aa7b8e69be342b8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43975 } } peers { permanent_uuid: "5752b93fbf53475cb5cf0593fd908fed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41427 } } peers { permanent_uuid: "ae19c5cb5e6d4abc8d5bf1463c817642" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45825 } }
I20260927 12:46:36.906293 7786 leader_election.cc:288] T efc617f1ea154360b636e3dddcd1fed1 P 5752b93fbf53475cb5cf0593fd908fed [CANDIDATE]: Term 1 election: Requested vote from peers 9387b2fa57714be0aa7b8e69be342b8e (127.0.98.1:43975), ae19c5cb5e6d4abc8d5bf1463c817642 (127.0.98.3:45825)
I20260927 12:46:36.906626 7699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc617f1ea154360b636e3dddcd1fed1" candidate_uuid: "5752b93fbf53475cb5cf0593fd908fed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae19c5cb5e6d4abc8d5bf1463c817642"
I20260927 12:46:36.906627 7426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc617f1ea154360b636e3dddcd1fed1" candidate_uuid: "5752b93fbf53475cb5cf0593fd908fed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9387b2fa57714be0aa7b8e69be342b8e"
I20260927 12:46:36.906756 7426 raft_consensus.cc:3037] T efc617f1ea154360b636e3dddcd1fed1 P 9387b2fa57714be0aa7b8e69be342b8e [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:46:36.906756 7699 raft_consensus.cc:3037] T efc617f1ea154360b636e3dddcd1fed1 P ae19c5cb5e6d4abc8d5bf1463c817642 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:46:36.908088 7426 raft_consensus.cc:2454] T efc617f1ea154360b636e3dddcd1fed1 P 9387b2fa57714be0aa7b8e69be342b8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5752b93fbf53475cb5cf0593fd908fed in term 1.
I20260927 12:46:36.908088 7699 raft_consensus.cc:2454] T efc617f1ea154360b636e3dddcd1fed1 P ae19c5cb5e6d4abc8d5bf1463c817642 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5752b93fbf53475cb5cf0593fd908fed in term 1.
I20260927 12:46:36.908368 7497 leader_election.cc:302] T efc617f1ea154360b636e3dddcd1fed1 P 5752b93fbf53475cb5cf0593fd908fed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5752b93fbf53475cb5cf0593fd908fed, ae19c5cb5e6d4abc8d5bf1463c817642; no voters:
I20260927 12:46:36.908546 7786 raft_consensus.cc:2781] T efc617f1ea154360b636e3dddcd1fed1 P 5752b93fbf53475cb5cf0593fd908fed [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:46:36.908823 7786 raft_consensus.cc:686] T efc617f1ea154360b636e3dddcd1fed1 P 5752b93fbf53475cb5cf0593fd908fed [term 1 LEADER]: Becoming Leader. State: Replica: 5752b93fbf53475cb5cf0593fd908fed, State: Running, Role: LEADER
I20260927 12:46:36.909353 7786 consensus_queue.cc:237] T efc617f1ea154360b636e3dddcd1fed1 P 5752b93fbf53475cb5cf0593fd908fed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387b2fa57714be0aa7b8e69be342b8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43975 } } peers { permanent_uuid: "5752b93fbf53475cb5cf0593fd908fed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41427 } } peers { permanent_uuid: "ae19c5cb5e6d4abc8d5bf1463c817642" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45825 } }
I20260927 12:46:36.918624 6509 catalog_manager.cc:5495] T efc617f1ea154360b636e3dddcd1fed1 P 5752b93fbf53475cb5cf0593fd908fed reported cstate change: term changed from 0 to 1, leader changed from <none> to 5752b93fbf53475cb5cf0593fd908fed (127.0.98.2). New cstate: current_term: 1 leader_uuid: "5752b93fbf53475cb5cf0593fd908fed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387b2fa57714be0aa7b8e69be342b8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5752b93fbf53475cb5cf0593fd908fed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae19c5cb5e6d4abc8d5bf1463c817642" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45825 } health_report { overall_health: UNKNOWN } } }
I20260927 12:46:36.940217 7775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:36.936536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43008 (local address 127.0.98.2:41427)
0927 12:46:36.936665 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:36.936668 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:36.936683 (+ 15us) server_negotiation.cc:407] Connection header received
0927 12:46:36.936860 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:36.936863 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:36.936923 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:36.937011 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:46:36.937264 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.937862 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.938821 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.939030 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.939106 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:36.939376 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:46:36.939815 (+ 439us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:46:36.939899 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:36.940003 (+ 104us) server_negotiation.cc:299] Negotiation successful
0927 12:46:36.940054 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 12:46:36.946166 7426 raft_consensus.cc:1261] T efc617f1ea154360b636e3dddcd1fed1 P 9387b2fa57714be0aa7b8e69be342b8e [term 1 FOLLOWER]: Refusing update from remote peer 5752b93fbf53475cb5cf0593fd908fed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:46:36.946166 7699 raft_consensus.cc:1261] T efc617f1ea154360b636e3dddcd1fed1 P ae19c5cb5e6d4abc8d5bf1463c817642 [term 1 FOLLOWER]: Refusing update from remote peer 5752b93fbf53475cb5cf0593fd908fed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:46:36.946902 7787 consensus_queue.cc:1035] T efc617f1ea154360b636e3dddcd1fed1 P 5752b93fbf53475cb5cf0593fd908fed [LEADER]: Connected to new peer: Peer: permanent_uuid: "9387b2fa57714be0aa7b8e69be342b8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:46:36.947098 7786 consensus_queue.cc:1035] T efc617f1ea154360b636e3dddcd1fed1 P 5752b93fbf53475cb5cf0593fd908fed [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae19c5cb5e6d4abc8d5bf1463c817642" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:46:36.959528 7791 mvcc.cc:205] Tried to move back new op lower bound from 7333942054684119040 to 7333942054543577088. Current Snapshot: MvccSnapshot[applied={T|T < 7333942054684119040}]
I20260927 12:46:36.964903 6507 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:46:36.965055 6507 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59146:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:46:36.969095 6507 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 205c8e3f55144799958e82a465ac9b66: Sending DeleteTablet for 3 replicas of tablet efc617f1ea154360b636e3dddcd1fed1
I20260927 12:46:36.970522 7679 tablet_service.cc:1514] Processing DeleteTablet for tablet efc617f1ea154360b636e3dddcd1fed1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:46:36 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45008
I20260927 12:46:36.971364 7543 tablet_service.cc:1514] Processing DeleteTablet for tablet efc617f1ea154360b636e3dddcd1fed1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:46:36 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42992
I20260927 12:46:36.971860 7801 tablet_replica.cc:330] T efc617f1ea154360b636e3dddcd1fed1 P 5752b93fbf53475cb5cf0593fd908fed: stopping tablet replica
I20260927 12:46:36.972254 7801 raft_consensus.cc:2229] T efc617f1ea154360b636e3dddcd1fed1 P 5752b93fbf53475cb5cf0593fd908fed [term 1 LEADER]: Raft consensus shutting down.
I20260927 12:46:36.972756 7801 raft_consensus.cc:2258] T efc617f1ea154360b636e3dddcd1fed1 P 5752b93fbf53475cb5cf0593fd908fed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:46:36.973023 7406 tablet_service.cc:1514] Processing DeleteTablet for tablet efc617f1ea154360b636e3dddcd1fed1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:46:36 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39298
I20260927 12:46:36.973951 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 6477
I20260927 12:46:36.975126 7801 ts_tablet_manager.cc:1905] T efc617f1ea154360b636e3dddcd1fed1 P 5752b93fbf53475cb5cf0593fd908fed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:46:36.978160 7801 ts_tablet_manager.cc:1918] T efc617f1ea154360b636e3dddcd1fed1 P 5752b93fbf53475cb5cf0593fd908fed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:46:36.978343 7801 log.cc:1196] T efc617f1ea154360b636e3dddcd1fed1 P 5752b93fbf53475cb5cf0593fd908fed: Deleting WAL directory at /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-1/wal/wals/efc617f1ea154360b636e3dddcd1fed1
I20260927 12:46:36.978861 7801 ts_tablet_manager.cc:1939] T efc617f1ea154360b636e3dddcd1fed1 P 5752b93fbf53475cb5cf0593fd908fed: Deleting consensus metadata
I20260927 12:46:36.981204 7761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:36.970975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48124 (local address 127.0.98.61:41239)
0927 12:46:36.971078 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:36.971082 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:36.971095 (+ 13us) server_negotiation.cc:407] Connection header received
0927 12:46:36.971132 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:36.971135 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:36.971181 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:36.971256 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:36.972155 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.972892 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.974013 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.974785 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.974864 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:36.977534 (+ 2670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:36.977538 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:36.977541 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:36.977570 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:36.979428 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:36.979943 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:36.979947 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:36.979948 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:36.980039 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:36.980384 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:36.980388 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:36.980389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:36.980575 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:36.980694 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:36.980928 (+ 234us) server_negotiation.cc:299] Negotiation successful
0927 12:46:36.981048 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
W20260927 12:46:36.981734 7499 connection.cc:535] server connection from 127.0.0.1:42992 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260927 12:46:36.982507 7278 connection.cc:535] server connection from 127.0.0.1:40302 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 12:46:36.985379 7800 tablet_replica.cc:330] T efc617f1ea154360b636e3dddcd1fed1 P ae19c5cb5e6d4abc8d5bf1463c817642: stopping tablet replica
I20260927 12:46:36.985756 7800 raft_consensus.cc:2229] T efc617f1ea154360b636e3dddcd1fed1 P ae19c5cb5e6d4abc8d5bf1463c817642 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:46:36.986334 7800 raft_consensus.cc:2258] T efc617f1ea154360b636e3dddcd1fed1 P ae19c5cb5e6d4abc8d5bf1463c817642 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:46:36.986644 7802 tablet_replica.cc:330] T efc617f1ea154360b636e3dddcd1fed1 P 9387b2fa57714be0aa7b8e69be342b8e: stopping tablet replica
I20260927 12:46:36.986953 7802 raft_consensus.cc:2229] T efc617f1ea154360b636e3dddcd1fed1 P 9387b2fa57714be0aa7b8e69be342b8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:46:36.987401 7802 raft_consensus.cc:2258] T efc617f1ea154360b636e3dddcd1fed1 P 9387b2fa57714be0aa7b8e69be342b8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:46:36.988744 7800 ts_tablet_manager.cc:1905] T efc617f1ea154360b636e3dddcd1fed1 P ae19c5cb5e6d4abc8d5bf1463c817642: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:46:36.990212 7759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:36.971194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40318 (local address 127.0.98.60:45213)
0927 12:46:36.971293 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:36.971297 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:36.981392 (+ 10095us) server_negotiation.cc:407] Connection header received
0927 12:46:36.981552 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:36.981556 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:36.981614 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:36.981664 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:36.982858 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.983573 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.984580 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.984895 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.984993 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:36.986566 (+ 1573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:36.986570 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:36.986573 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:36.986600 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:36.988248 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:36.988765 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:36.988769 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:36.988771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:36.988872 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:36.989205 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:36.989209 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:36.989211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:36.989445 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:36.989560 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:36.989883 (+ 323us) server_negotiation.cc:299] Negotiation successful
0927 12:46:36.990009 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 12:46:36.991493 7800 ts_tablet_manager.cc:1918] T efc617f1ea154360b636e3dddcd1fed1 P ae19c5cb5e6d4abc8d5bf1463c817642: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:46:36.991592 7800 log.cc:1196] T efc617f1ea154360b636e3dddcd1fed1 P ae19c5cb5e6d4abc8d5bf1463c817642: Deleting WAL directory at /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-2/wal/wals/efc617f1ea154360b636e3dddcd1fed1
I20260927 12:46:36.991884 7800 ts_tablet_manager.cc:1939] T efc617f1ea154360b636e3dddcd1fed1 P ae19c5cb5e6d4abc8d5bf1463c817642: Deleting consensus metadata
I20260927 12:46:36.994696 7802 ts_tablet_manager.cc:1905] T efc617f1ea154360b636e3dddcd1fed1 P 9387b2fa57714be0aa7b8e69be342b8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:46:36.997951 7802 ts_tablet_manager.cc:1918] T efc617f1ea154360b636e3dddcd1fed1 P 9387b2fa57714be0aa7b8e69be342b8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:46:36.998085 7802 log.cc:1196] T efc617f1ea154360b636e3dddcd1fed1 P 9387b2fa57714be0aa7b8e69be342b8e: Deleting WAL directory at /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-0/wal/wals/efc617f1ea154360b636e3dddcd1fed1
I20260927 12:46:36.998416 7802 ts_tablet_manager.cc:1939] T efc617f1ea154360b636e3dddcd1fed1 P 9387b2fa57714be0aa7b8e69be342b8e: Deleting consensus metadata
W20260927 12:46:36.999583 7359 connection.cc:441] server connection from 127.0.0.1:39298 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39298 (request call id 1) could send its response
W20260927 12:46:37.000011 7634 connection.cc:441] server connection from 127.0.0.1:45008 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:45008 (request call id 1) could send its response
I20260927 12:46:37.001041 7761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:36.986208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48136 (local address 127.0.98.61:41239)
0927 12:46:36.986323 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:36.986327 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:36.988697 (+ 2370us) server_negotiation.cc:407] Connection header received
0927 12:46:36.988875 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:36.988878 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:36.988928 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:36.988977 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:36.989805 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.990575 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.991602 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.991905 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.991983 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:36.996739 (+ 4756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:36.996744 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:36.996746 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:36.996776 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:36.998631 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:36.999463 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:36.999467 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:36.999469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:36.999561 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:37.000109 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:37.000113 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:37.000115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:37.000352 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:37.000454 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:37.000780 (+ 326us) server_negotiation.cc:299] Negotiation successful
0927 12:46:37.000887 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 12:46:37.002261 7476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:36.986451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40322 (local address 127.0.98.60:45213)
0927 12:46:36.986562 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:36.986566 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:36.992634 (+ 6068us) server_negotiation.cc:407] Connection header received
0927 12:46:36.992674 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:36.992677 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:36.992723 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:36.992805 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:36.994889 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.995632 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.996776 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:36.997080 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:36.997152 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:36.998980 (+ 1828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:36.998983 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:36.998986 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:36.999012 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:37.000492 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:37.000974 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:37.000978 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:37.000980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:37.001063 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:37.001353 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:37.001356 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:37.001358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:37.001550 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:37.001658 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:37.001976 (+ 318us) server_negotiation.cc:299] Negotiation successful
0927 12:46:37.002115 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 12:46:37.961668 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:37.961142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40735 (local address 127.0.98.1:54343)
0927 12:46:37.961459 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:37.961554 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":76,"threads_started":1}
I20260927 12:46:37.961668 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:37.961143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40735 (local address 127.0.98.3:33709)
0927 12:46:37.961459 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:37.961554 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":75,"threads_started":1}
W20260927 12:46:37.962002 7746 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40735 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:40735: connect: Connection refused (error 111)
W20260927 12:46:37.961988 7473 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40735 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:40735: connect: Connection refused (error 111)
I20260927 12:46:37.963949 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:37.963529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40735 (local address 127.0.98.2:48415)
0927 12:46:37.963761 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:37.963847 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":69,"threads_started":1}
W20260927 12:46:37.964320 7610 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40735 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:40735: connect: Connection refused (error 111)
I20260927 12:46:38.470358 7813 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4fdbce7f7c3d4fda8c562a4fb4072671 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 205c8e3f55144799958e82a465ac9b66)
I20260927 12:46:38.470558 7813 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4fdbce7f7c3d4fda8c562a4fb4072671 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "205c8e3f55144799958e82a465ac9b66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40735 } } peers { permanent_uuid: "4fdbce7f7c3d4fda8c562a4fb4072671" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41239 } attrs { promote: false } } peers { permanent_uuid: "70b88e0990ca471d93129ebffc98d9cf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45213 } attrs { promote: false } }
I20260927 12:46:38.470870 7814 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 205c8e3f55144799958e82a465ac9b66)
I20260927 12:46:38.471046 7814 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "205c8e3f55144799958e82a465ac9b66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40735 } } peers { permanent_uuid: "4fdbce7f7c3d4fda8c562a4fb4072671" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41239 } attrs { promote: false } } peers { permanent_uuid: "70b88e0990ca471d93129ebffc98d9cf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45213 } attrs { promote: false } }
I20260927 12:46:38.471596 7813 leader_election.cc:288] T 00000000000000000000000000000000 P 4fdbce7f7c3d4fda8c562a4fb4072671 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 205c8e3f55144799958e82a465ac9b66 (127.0.98.62:40735), 70b88e0990ca471d93129ebffc98d9cf (127.0.98.60:45213)
I20260927 12:46:38.471925 7814 leader_election.cc:288] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 205c8e3f55144799958e82a465ac9b66 (127.0.98.62:40735), 4fdbce7f7c3d4fda8c562a4fb4072671 (127.0.98.61:41239)
I20260927 12:46:38.472285 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:38.471791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40735 (local address 127.0.0.1:59678)
0927 12:46:38.472103 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:38.472181 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":74,"threads_started":1}
W20260927 12:46:38.472469 7095 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40735: connect: Connection refused (error 111)
I20260927 12:46:38.472790 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:38.472081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40735 (local address 127.0.0.1:59686)
0927 12:46:38.472631 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:38.472702 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":58,"threads_started":1}
W20260927 12:46:38.472945 7278 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40735: connect: Connection refused (error 111)
I20260927 12:46:38.473393 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:38.473160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40735 (local address 127.0.0.1:59694)
0927 12:46:38.473286 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:38.473332 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260927 12:46:38.473526 7095 leader_election.cc:334] T 00000000000000000000000000000000 P 4fdbce7f7c3d4fda8c562a4fb4072671 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 205c8e3f55144799958e82a465ac9b66 (127.0.98.62:40735): Network error: Client connection negotiation failed: client connection to 127.0.98.62:40735: connect: Connection refused (error 111)
I20260927 12:46:38.473815 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:38.473594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40735 (local address 127.0.0.1:59704)
0927 12:46:38.473721 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:38.473763 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260927 12:46:38.473935 7278 leader_election.cc:334] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 205c8e3f55144799958e82a465ac9b66 (127.0.98.62:40735): Network error: Client connection negotiation failed: client connection to 127.0.98.62:40735: connect: Connection refused (error 111)
I20260927 12:46:38.476905 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:38.472096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41239 (local address 127.0.0.1:48138)
0927 12:46:38.472646 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:38.472657 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:38.472766 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:38.473222 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:38.473225 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:38.473242 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:46:38.473336 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:38.473341 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:38.474387 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:38.474390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:38.475635 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:38.475645 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:38.476489 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:38.476493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:38.476588 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:38.476612 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 12:46:38.476697 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 12:46:38.476769 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":57,"threads_started":1}
I20260927 12:46:38.476979 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:38.472168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48138 (local address 127.0.98.61:41239)
0927 12:46:38.472478 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:38.472482 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:38.472706 (+ 224us) server_negotiation.cc:407] Connection header received
0927 12:46:38.472987 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:38.472990 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:38.473076 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:38.473126 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:46:38.473503 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:38.474303 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:38.475738 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:38.476374 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:38.476459 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:38.476510 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:38.476711 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 12:46:38.476841 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":72,"server-negotiator.queue_time_us":242,"thread_start_us":77,"threads_started":1}
I20260927 12:46:38.477006 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:38.472397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45213 (local address 127.0.0.1:40326)
0927 12:46:38.472566 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:38.472578 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:38.472690 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:38.473030 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:38.473033 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:38.473068 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:46:38.473203 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:38.473210 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:38.474269 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:38.474271 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:38.475541 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:38.475552 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:38.476489 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:38.476493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:38.476588 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:38.476612 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 12:46:38.476697 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 12:46:38.476861 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":1,"spinlock_wait_cycles":22272,"thread_start_us":43,"threads_started":1}
I20260927 12:46:38.476958 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:38.472256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40326 (local address 127.0.98.60:45213)
0927 12:46:38.472500 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:38.472504 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:38.472783 (+ 279us) server_negotiation.cc:407] Connection header received
0927 12:46:38.472827 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:38.472829 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:38.472866 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:38.472928 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:46:38.473306 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:38.474155 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:38.475655 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:38.476348 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:38.476453 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:38.476511 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:38.476711 (+ 200us) server_negotiation.cc:299] Negotiation successful
0927 12:46:38.476836 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":185,"thread_start_us":40,"threads_started":1}
I20260927 12:46:38.477698 7294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4fdbce7f7c3d4fda8c562a4fb4072671" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "70b88e0990ca471d93129ebffc98d9cf" is_pre_election: true
I20260927 12:46:38.477917 7294 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fdbce7f7c3d4fda8c562a4fb4072671 in term 1.
I20260927 12:46:38.477913 7119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "70b88e0990ca471d93129ebffc98d9cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4fdbce7f7c3d4fda8c562a4fb4072671" is_pre_election: true
I20260927 12:46:38.478101 7119 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4fdbce7f7c3d4fda8c562a4fb4072671 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70b88e0990ca471d93129ebffc98d9cf in term 1.
I20260927 12:46:38.478263 7094 leader_election.cc:302] T 00000000000000000000000000000000 P 4fdbce7f7c3d4fda8c562a4fb4072671 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fdbce7f7c3d4fda8c562a4fb4072671, 70b88e0990ca471d93129ebffc98d9cf; no voters: 205c8e3f55144799958e82a465ac9b66
I20260927 12:46:38.478420 7277 leader_election.cc:302] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fdbce7f7c3d4fda8c562a4fb4072671, 70b88e0990ca471d93129ebffc98d9cf; no voters: 205c8e3f55144799958e82a465ac9b66
I20260927 12:46:38.478464 7813 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4fdbce7f7c3d4fda8c562a4fb4072671 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 12:46:38.478510 7813 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4fdbce7f7c3d4fda8c562a4fb4072671 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 205c8e3f55144799958e82a465ac9b66)
I20260927 12:46:38.478551 7813 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fdbce7f7c3d4fda8c562a4fb4072671 [term 1 FOLLOWER]: Advancing to term 2
I20260927 12:46:38.478612 7814 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 12:46:38.478659 7814 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf [term 1 FOLLOWER]: Starting leader election (detected failure of leader 205c8e3f55144799958e82a465ac9b66)
I20260927 12:46:38.478694 7814 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf [term 1 FOLLOWER]: Advancing to term 2
I20260927 12:46:38.479578 7813 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4fdbce7f7c3d4fda8c562a4fb4072671 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "205c8e3f55144799958e82a465ac9b66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40735 } } peers { permanent_uuid: "4fdbce7f7c3d4fda8c562a4fb4072671" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41239 } attrs { promote: false } } peers { permanent_uuid: "70b88e0990ca471d93129ebffc98d9cf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45213 } attrs { promote: false } }
I20260927 12:46:38.479694 7814 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "205c8e3f55144799958e82a465ac9b66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40735 } } peers { permanent_uuid: "4fdbce7f7c3d4fda8c562a4fb4072671" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41239 } attrs { promote: false } } peers { permanent_uuid: "70b88e0990ca471d93129ebffc98d9cf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45213 } attrs { promote: false } }
I20260927 12:46:38.479872 7813 leader_election.cc:288] T 00000000000000000000000000000000 P 4fdbce7f7c3d4fda8c562a4fb4072671 [CANDIDATE]: Term 2 election: Requested vote from peers 205c8e3f55144799958e82a465ac9b66 (127.0.98.62:40735), 70b88e0990ca471d93129ebffc98d9cf (127.0.98.60:45213)
I20260927 12:46:38.480021 7814 leader_election.cc:288] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf [CANDIDATE]: Term 2 election: Requested vote from peers 205c8e3f55144799958e82a465ac9b66 (127.0.98.62:40735), 4fdbce7f7c3d4fda8c562a4fb4072671 (127.0.98.61:41239)
I20260927 12:46:38.480180 7294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4fdbce7f7c3d4fda8c562a4fb4072671" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "70b88e0990ca471d93129ebffc98d9cf"
I20260927 12:46:38.480342 7294 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 4fdbce7f7c3d4fda8c562a4fb4072671 in current term 2: Already voted for candidate 70b88e0990ca471d93129ebffc98d9cf in this term.
I20260927 12:46:38.480356 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:38.480102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40735 (local address 127.0.0.1:59714)
0927 12:46:38.480239 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:38.480290 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260927 12:46:38.480412 7119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "70b88e0990ca471d93129ebffc98d9cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4fdbce7f7c3d4fda8c562a4fb4072671"
I20260927 12:46:38.480554 7119 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 4fdbce7f7c3d4fda8c562a4fb4072671 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 70b88e0990ca471d93129ebffc98d9cf in current term 2: Already voted for candidate 4fdbce7f7c3d4fda8c562a4fb4072671 in this term.
I20260927 12:46:38.480995 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:38.480749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40735 (local address 127.0.0.1:59722)
0927 12:46:38.480895 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:38.480941 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
W20260927 12:46:38.481087 7278 leader_election.cc:334] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 205c8e3f55144799958e82a465ac9b66 (127.0.98.62:40735): Network error: Client connection negotiation failed: client connection to 127.0.98.62:40735: connect: Connection refused (error 111)
I20260927 12:46:38.481113 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:38.480011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40735 (local address 127.0.0.1:59712)
0927 12:46:38.480301 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:38.480357 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184}
I20260927 12:46:38.481149 7278 leader_election.cc:302] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 70b88e0990ca471d93129ebffc98d9cf; no voters: 205c8e3f55144799958e82a465ac9b66, 4fdbce7f7c3d4fda8c562a4fb4072671
I20260927 12:46:38.481266 7814 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 12:46:38.481663 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:38.481461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40735 (local address 127.0.0.1:59734)
0927 12:46:38.481576 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:38.481618 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260927 12:46:38.481761 7095 leader_election.cc:334] T 00000000000000000000000000000000 P 4fdbce7f7c3d4fda8c562a4fb4072671 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 205c8e3f55144799958e82a465ac9b66 (127.0.98.62:40735): Network error: Client connection negotiation failed: client connection to 127.0.98.62:40735: connect: Connection refused (error 111)
I20260927 12:46:38.481828 7095 leader_election.cc:302] T 00000000000000000000000000000000 P 4fdbce7f7c3d4fda8c562a4fb4072671 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fdbce7f7c3d4fda8c562a4fb4072671; no voters: 205c8e3f55144799958e82a465ac9b66, 70b88e0990ca471d93129ebffc98d9cf
I20260927 12:46:38.481945 7813 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4fdbce7f7c3d4fda8c562a4fb4072671 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 12:46:38.962987 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:38.962550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40735 (local address 127.0.98.1:52549)
0927 12:46:38.962821 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:38.962890 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":96,"threads_started":1}
I20260927 12:46:38.963040 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:38.962553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40735 (local address 127.0.98.3:47507)
0927 12:46:38.962888 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:38.962948 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":101,"threads_started":1}
I20260927 12:46:38.965216 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:38.964856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40735 (local address 127.0.98.2:39821)
0927 12:46:38.965079 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:38.965133 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":73,"threads_started":1}
I20260927 12:46:39.964128 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:39.963681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40735 (local address 127.0.98.1:58061)
0927 12:46:39.963982 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:39.964038 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":77,"threads_started":1}
I20260927 12:46:39.964469 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:39.963841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40735 (local address 127.0.98.3:45535)
0927 12:46:39.964123 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:39.964378 (+ 255us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":66,"threads_started":1}
I20260927 12:46:39.966266 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:39.965864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40735 (local address 127.0.98.2:47775)
0927 12:46:39.966136 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:39.966188 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":66,"threads_started":1}
I20260927 12:46:40.263518 7829 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:46:40.263665 7829 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "205c8e3f55144799958e82a465ac9b66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40735 } } peers { permanent_uuid: "4fdbce7f7c3d4fda8c562a4fb4072671" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41239 } attrs { promote: false } } peers { permanent_uuid: "70b88e0990ca471d93129ebffc98d9cf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45213 } attrs { promote: false } }
I20260927 12:46:40.264124 7829 leader_election.cc:288] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 205c8e3f55144799958e82a465ac9b66 (127.0.98.62:40735), 4fdbce7f7c3d4fda8c562a4fb4072671 (127.0.98.61:41239)
I20260927 12:46:40.264583 7119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "70b88e0990ca471d93129ebffc98d9cf" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4fdbce7f7c3d4fda8c562a4fb4072671" is_pre_election: true
I20260927 12:46:40.264722 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:40.264344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40735 (local address 127.0.0.1:59944)
0927 12:46:40.264559 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:40.264618 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":59,"threads_started":1}
I20260927 12:46:40.264729 7119 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4fdbce7f7c3d4fda8c562a4fb4072671 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70b88e0990ca471d93129ebffc98d9cf in term 2.
I20260927 12:46:40.265031 7277 leader_election.cc:302] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fdbce7f7c3d4fda8c562a4fb4072671, 70b88e0990ca471d93129ebffc98d9cf; no voters:
I20260927 12:46:40.265205 7829 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 12:46:40.265266 7829 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:46:40.265304 7829 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf [term 2 FOLLOWER]: Advancing to term 3
I20260927 12:46:40.265548 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:40.265309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40735 (local address 127.0.0.1:59956)
0927 12:46:40.265445 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:40.265490 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260927 12:46:40.265671 7278 leader_election.cc:334] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 205c8e3f55144799958e82a465ac9b66 (127.0.98.62:40735): Network error: Client connection negotiation failed: client connection to 127.0.98.62:40735: connect: Connection refused (error 111)
I20260927 12:46:40.266492 7829 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "205c8e3f55144799958e82a465ac9b66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40735 } } peers { permanent_uuid: "4fdbce7f7c3d4fda8c562a4fb4072671" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41239 } attrs { promote: false } } peers { permanent_uuid: "70b88e0990ca471d93129ebffc98d9cf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45213 } attrs { promote: false } }
I20260927 12:46:40.266803 7829 leader_election.cc:288] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf [CANDIDATE]: Term 3 election: Requested vote from peers 205c8e3f55144799958e82a465ac9b66 (127.0.98.62:40735), 4fdbce7f7c3d4fda8c562a4fb4072671 (127.0.98.61:41239)
I20260927 12:46:40.267200 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:40.266992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40735 (local address 127.0.0.1:59960)
0927 12:46:40.267097 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:40.267146 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 12:46:40.267185 7119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "70b88e0990ca471d93129ebffc98d9cf" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4fdbce7f7c3d4fda8c562a4fb4072671"
I20260927 12:46:40.267294 7119 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fdbce7f7c3d4fda8c562a4fb4072671 [term 2 FOLLOWER]: Advancing to term 3
I20260927 12:46:40.267863 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:40.267633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40735 (local address 127.0.0.1:59976)
0927 12:46:40.267780 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:40.267820 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260927 12:46:40.267956 7278 leader_election.cc:334] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 205c8e3f55144799958e82a465ac9b66 (127.0.98.62:40735): Network error: Client connection negotiation failed: client connection to 127.0.98.62:40735: connect: Connection refused (error 111)
I20260927 12:46:40.268357 7119 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4fdbce7f7c3d4fda8c562a4fb4072671 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70b88e0990ca471d93129ebffc98d9cf in term 3.
I20260927 12:46:40.268615 7277 leader_election.cc:302] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fdbce7f7c3d4fda8c562a4fb4072671, 70b88e0990ca471d93129ebffc98d9cf; no voters: 205c8e3f55144799958e82a465ac9b66
I20260927 12:46:40.268769 7829 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf [term 3 FOLLOWER]: Leader election won for term 3
I20260927 12:46:40.268970 7829 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf [term 3 LEADER]: Becoming Leader. State: Replica: 70b88e0990ca471d93129ebffc98d9cf, State: Running, Role: LEADER
I20260927 12:46:40.269207 7829 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "205c8e3f55144799958e82a465ac9b66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40735 } } peers { permanent_uuid: "4fdbce7f7c3d4fda8c562a4fb4072671" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41239 } attrs { promote: false } } peers { permanent_uuid: "70b88e0990ca471d93129ebffc98d9cf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45213 } attrs { promote: false } }
I20260927 12:46:40.270637 7832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70b88e0990ca471d93129ebffc98d9cf. Latest consensus state: current_term: 3 leader_uuid: "70b88e0990ca471d93129ebffc98d9cf" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "205c8e3f55144799958e82a465ac9b66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40735 } } peers { permanent_uuid: "4fdbce7f7c3d4fda8c562a4fb4072671" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41239 } attrs { promote: false } } peers { permanent_uuid: "70b88e0990ca471d93129ebffc98d9cf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45213 } attrs { promote: false } } }
I20260927 12:46:40.270808 7832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf [sys.catalog]: This master's current role is: LEADER
I20260927 12:46:40.274241 7834 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:46:40.276670 7834 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:46:40.277035 7834 catalog_manager.cc:1259] Loaded cluster ID: ee03576c83fa435283af71c4b0841e2f
I20260927 12:46:40.277081 7834 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:46:40.277586 7834 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:46:40.277922 7834 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf: Loaded TSK: 0
I20260927 12:46:40.278595 7834 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:46:40.293479 7285 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40322:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:46:40.296785 7285 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:46:40.300120 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:40.299851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40735 (local address 127.0.0.1:59998)
0927 12:46:40.299968 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:40.300031 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 12:46:40.300236 7119 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4fdbce7f7c3d4fda8c562a4fb4072671 [term 3 FOLLOWER]: Refusing update from remote peer 70b88e0990ca471d93129ebffc98d9cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260927 12:46:40.300662 7832 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fdbce7f7c3d4fda8c562a4fb4072671" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41239 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260927 12:46:40.300879 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:40.300620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40735 (local address 127.0.0.1:60002)
0927 12:46:40.300720 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:40.300762 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 12:46:40.300980 7278 consensus_peers.cc:487] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf -> Peer 205c8e3f55144799958e82a465ac9b66 (127.0.98.62:40735): Couldn't send request to peer 205c8e3f55144799958e82a465ac9b66. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40735: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 12:46:40.302174 7836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fdbce7f7c3d4fda8c562a4fb4072671 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70b88e0990ca471d93129ebffc98d9cf. Latest consensus state: current_term: 3 leader_uuid: "70b88e0990ca471d93129ebffc98d9cf" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "205c8e3f55144799958e82a465ac9b66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40735 } } peers { permanent_uuid: "4fdbce7f7c3d4fda8c562a4fb4072671" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41239 } attrs { promote: false } } peers { permanent_uuid: "70b88e0990ca471d93129ebffc98d9cf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45213 } attrs { promote: false } } }
I20260927 12:46:40.302318 7836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fdbce7f7c3d4fda8c562a4fb4072671 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:46:40.303434 7829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "70b88e0990ca471d93129ebffc98d9cf" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "205c8e3f55144799958e82a465ac9b66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40735 } } peers { permanent_uuid: "4fdbce7f7c3d4fda8c562a4fb4072671" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41239 } attrs { promote: false } } peers { permanent_uuid: "70b88e0990ca471d93129ebffc98d9cf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45213 } attrs { promote: false } } }
I20260927 12:46:40.303548 7829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf [sys.catalog]: This master's current role is: LEADER
I20260927 12:46:40.303857 7836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fdbce7f7c3d4fda8c562a4fb4072671 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "70b88e0990ca471d93129ebffc98d9cf" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "205c8e3f55144799958e82a465ac9b66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40735 } } peers { permanent_uuid: "4fdbce7f7c3d4fda8c562a4fb4072671" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41239 } attrs { promote: false } } peers { permanent_uuid: "70b88e0990ca471d93129ebffc98d9cf" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45213 } attrs { promote: false } } }
I20260927 12:46:40.303972 7836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fdbce7f7c3d4fda8c562a4fb4072671 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:46:40.314148 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:40.309395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43975 (local address 127.0.0.1:39314)
0927 12:46:40.309789 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:40.309803 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:40.309882 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:40.310738 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:40.310741 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:40.310758 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:46:40.310848 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:40.310854 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:40.311808 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:40.311812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:40.312913 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:40.312922 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:40.313745 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:40.313749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:40.313835 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:40.313855 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 12:46:40.313911 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 12:46:40.313974 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"mutex_wait_us":75}
I20260927 12:46:40.314189 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:40.309514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39314 (local address 127.0.98.1:43975)
0927 12:46:40.310526 (+ 1012us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:40.310530 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:40.310542 (+ 12us) server_negotiation.cc:407] Connection header received
0927 12:46:40.310588 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:40.310590 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:40.310632 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:40.310714 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:46:40.310952 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:40.311663 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:40.313042 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:40.313636 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:40.313737 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:40.313805 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:40.313944 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 12:46:40.314019 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":937,"thread_start_us":95,"threads_started":1}
I20260927 12:46:40.314491 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:40.309969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43024 (local address 127.0.98.2:41427)
0927 12:46:40.310337 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:40.310341 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:40.310355 (+ 14us) server_negotiation.cc:407] Connection header received
0927 12:46:40.310409 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:40.310411 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:40.310457 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:40.310546 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:46:40.310797 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:40.311662 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:40.313291 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:40.313902 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:40.314008 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:40.314091 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:40.314266 (+ 175us) server_negotiation.cc:299] Negotiation successful
0927 12:46:40.314359 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":83,"threads_started":1}
I20260927 12:46:40.315048 7406 tablet_service.cc:1467] Processing CreateTablet for tablet 3fe884d6b720451b952e0ae2082fa39d (DEFAULT_TABLE table=test-table [id=b9c46bdbcf4b46ada905e569b2ee53a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:46:40.315112 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:40.309845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41427 (local address 127.0.0.1:43024)
0927 12:46:40.310068 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:40.310080 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:40.310148 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:40.310548 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:40.310550 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:40.310565 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:46:40.310668 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:40.310674 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:40.311840 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:40.311843 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:40.313169 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:40.313178 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:40.314015 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:40.314019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:40.314124 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:40.314144 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 12:46:40.314877 (+ 733us) client_negotiation.cc:240] Negotiation successful
0927 12:46:40.314990 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":62,"thread_start_us":92,"threads_started":1}
I20260927 12:46:40.315253 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:40.310148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45825 (local address 127.0.0.1:45020)
0927 12:46:40.310342 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:40.310353 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:40.310423 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:40.311007 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:40.311009 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:40.311023 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:46:40.311109 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:40.311114 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:40.311840 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:40.311843 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:40.313169 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:40.313178 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:40.314788 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:40.314791 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:40.314893 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:40.314953 (+ 60us) client_negotiation.cc:769] Sending connection context
0927 12:46:40.315013 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 12:46:40.315086 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":127,"thread_start_us":83,"threads_started":1}
I20260927 12:46:40.315261 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:40.310467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45020 (local address 127.0.98.3:45825)
0927 12:46:40.310818 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:40.310823 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:40.310836 (+ 13us) server_negotiation.cc:407] Connection header received
0927 12:46:40.310882 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:40.310885 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:40.310930 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:40.311003 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:46:40.311205 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:40.311752 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:40.314154 (+ 2402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:40.314677 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:40.314788 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:40.314849 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:40.315039 (+ 190us) server_negotiation.cc:299] Negotiation successful
0927 12:46:40.315110 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":74,"threads_started":1}
I20260927 12:46:40.315407 7406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fe884d6b720451b952e0ae2082fa39d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:46:40.315631 7543 tablet_service.cc:1467] Processing CreateTablet for tablet 3fe884d6b720451b952e0ae2082fa39d (DEFAULT_TABLE table=test-table [id=b9c46bdbcf4b46ada905e569b2ee53a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:46:40.315893 7543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fe884d6b720451b952e0ae2082fa39d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:46:40.315900 7679 tablet_service.cc:1467] Processing CreateTablet for tablet 3fe884d6b720451b952e0ae2082fa39d (DEFAULT_TABLE table=test-table [id=b9c46bdbcf4b46ada905e569b2ee53a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:46:40.316121 7679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fe884d6b720451b952e0ae2082fa39d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:46:40.317703 7847 tablet_bootstrap.cc:492] T 3fe884d6b720451b952e0ae2082fa39d P 9387b2fa57714be0aa7b8e69be342b8e: Bootstrap starting.
I20260927 12:46:40.318655 7848 tablet_bootstrap.cc:492] T 3fe884d6b720451b952e0ae2082fa39d P 5752b93fbf53475cb5cf0593fd908fed: Bootstrap starting.
I20260927 12:46:40.318720 7847 tablet_bootstrap.cc:654] T 3fe884d6b720451b952e0ae2082fa39d P 9387b2fa57714be0aa7b8e69be342b8e: Neither blocks nor log segments found. Creating new log.
I20260927 12:46:40.319515 7848 tablet_bootstrap.cc:654] T 3fe884d6b720451b952e0ae2082fa39d P 5752b93fbf53475cb5cf0593fd908fed: Neither blocks nor log segments found. Creating new log.
I20260927 12:46:40.320031 7847 tablet_bootstrap.cc:492] T 3fe884d6b720451b952e0ae2082fa39d P 9387b2fa57714be0aa7b8e69be342b8e: No bootstrap required, opened a new log
I20260927 12:46:40.320117 7849 tablet_bootstrap.cc:492] T 3fe884d6b720451b952e0ae2082fa39d P ae19c5cb5e6d4abc8d5bf1463c817642: Bootstrap starting.
I20260927 12:46:40.320189 7847 ts_tablet_manager.cc:1397] T 3fe884d6b720451b952e0ae2082fa39d P 9387b2fa57714be0aa7b8e69be342b8e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 12:46:40.320668 7847 raft_consensus.cc:348] T 3fe884d6b720451b952e0ae2082fa39d P 9387b2fa57714be0aa7b8e69be342b8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387b2fa57714be0aa7b8e69be342b8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43975 } } peers { permanent_uuid: "5752b93fbf53475cb5cf0593fd908fed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41427 } } peers { permanent_uuid: "ae19c5cb5e6d4abc8d5bf1463c817642" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45825 } }
I20260927 12:46:40.320777 7847 raft_consensus.cc:374] T 3fe884d6b720451b952e0ae2082fa39d P 9387b2fa57714be0aa7b8e69be342b8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:46:40.320811 7847 raft_consensus.cc:729] T 3fe884d6b720451b952e0ae2082fa39d P 9387b2fa57714be0aa7b8e69be342b8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9387b2fa57714be0aa7b8e69be342b8e, State: Initialized, Role: FOLLOWER
I20260927 12:46:40.320938 7849 tablet_bootstrap.cc:654] T 3fe884d6b720451b952e0ae2082fa39d P ae19c5cb5e6d4abc8d5bf1463c817642: Neither blocks nor log segments found. Creating new log.
I20260927 12:46:40.320947 7847 consensus_queue.cc:260] T 3fe884d6b720451b952e0ae2082fa39d P 9387b2fa57714be0aa7b8e69be342b8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387b2fa57714be0aa7b8e69be342b8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43975 } } peers { permanent_uuid: "5752b93fbf53475cb5cf0593fd908fed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41427 } } peers { permanent_uuid: "ae19c5cb5e6d4abc8d5bf1463c817642" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45825 } }
I20260927 12:46:40.321306 7847 ts_tablet_manager.cc:1428] T 3fe884d6b720451b952e0ae2082fa39d P 9387b2fa57714be0aa7b8e69be342b8e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 12:46:40.321843 7848 tablet_bootstrap.cc:492] T 3fe884d6b720451b952e0ae2082fa39d P 5752b93fbf53475cb5cf0593fd908fed: No bootstrap required, opened a new log
I20260927 12:46:40.321971 7848 ts_tablet_manager.cc:1397] T 3fe884d6b720451b952e0ae2082fa39d P 5752b93fbf53475cb5cf0593fd908fed: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 12:46:40.322398 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:40.321763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40735 (local address 127.0.98.1:45181)
0927 12:46:40.322258 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:40.322322 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418}
I20260927 12:46:40.322531 7848 raft_consensus.cc:348] T 3fe884d6b720451b952e0ae2082fa39d P 5752b93fbf53475cb5cf0593fd908fed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387b2fa57714be0aa7b8e69be342b8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43975 } } peers { permanent_uuid: "5752b93fbf53475cb5cf0593fd908fed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41427 } } peers { permanent_uuid: "ae19c5cb5e6d4abc8d5bf1463c817642" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45825 } }
I20260927 12:46:40.322690 7849 tablet_bootstrap.cc:492] T 3fe884d6b720451b952e0ae2082fa39d P ae19c5cb5e6d4abc8d5bf1463c817642: No bootstrap required, opened a new log
I20260927 12:46:40.322737 7848 raft_consensus.cc:374] T 3fe884d6b720451b952e0ae2082fa39d P 5752b93fbf53475cb5cf0593fd908fed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:46:40.322815 7849 ts_tablet_manager.cc:1397] T 3fe884d6b720451b952e0ae2082fa39d P ae19c5cb5e6d4abc8d5bf1463c817642: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 12:46:40.322845 7848 raft_consensus.cc:729] T 3fe884d6b720451b952e0ae2082fa39d P 5752b93fbf53475cb5cf0593fd908fed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5752b93fbf53475cb5cf0593fd908fed, State: Initialized, Role: FOLLOWER
I20260927 12:46:40.323038 7848 consensus_queue.cc:260] T 3fe884d6b720451b952e0ae2082fa39d P 5752b93fbf53475cb5cf0593fd908fed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387b2fa57714be0aa7b8e69be342b8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43975 } } peers { permanent_uuid: "5752b93fbf53475cb5cf0593fd908fed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41427 } } peers { permanent_uuid: "ae19c5cb5e6d4abc8d5bf1463c817642" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45825 } }
I20260927 12:46:40.323275 7849 raft_consensus.cc:348] T 3fe884d6b720451b952e0ae2082fa39d P ae19c5cb5e6d4abc8d5bf1463c817642 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387b2fa57714be0aa7b8e69be342b8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43975 } } peers { permanent_uuid: "5752b93fbf53475cb5cf0593fd908fed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41427 } } peers { permanent_uuid: "ae19c5cb5e6d4abc8d5bf1463c817642" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45825 } }
I20260927 12:46:40.323383 7849 raft_consensus.cc:374] T 3fe884d6b720451b952e0ae2082fa39d P ae19c5cb5e6d4abc8d5bf1463c817642 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:46:40.323457 7849 raft_consensus.cc:729] T 3fe884d6b720451b952e0ae2082fa39d P ae19c5cb5e6d4abc8d5bf1463c817642 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae19c5cb5e6d4abc8d5bf1463c817642, State: Initialized, Role: FOLLOWER
I20260927 12:46:40.323598 7848 ts_tablet_manager.cc:1428] T 3fe884d6b720451b952e0ae2082fa39d P 5752b93fbf53475cb5cf0593fd908fed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 12:46:40.323724 7472 heartbeater.cc:503] Master 127.0.98.60:45213 was elected leader, sending a full tablet report...
I20260927 12:46:40.323709 7849 consensus_queue.cc:260] T 3fe884d6b720451b952e0ae2082fa39d P ae19c5cb5e6d4abc8d5bf1463c817642 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387b2fa57714be0aa7b8e69be342b8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43975 } } peers { permanent_uuid: "5752b93fbf53475cb5cf0593fd908fed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41427 } } peers { permanent_uuid: "ae19c5cb5e6d4abc8d5bf1463c817642" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45825 } }
I20260927 12:46:40.324044 7849 ts_tablet_manager.cc:1428] T 3fe884d6b720451b952e0ae2082fa39d P ae19c5cb5e6d4abc8d5bf1463c817642: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 12:46:40.325006 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:40.324513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40735 (local address 127.0.98.3:52271)
0927 12:46:40.324856 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:40.324933 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270}
I20260927 12:46:40.325588 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:40.325369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40735 (local address 127.0.98.2:53285)
0927 12:46:40.325473 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:40.325523 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 12:46:40.325757 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:40.325520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40735 (local address 127.0.98.3:43895)
0927 12:46:40.325634 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:40.325690 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 12:46:40.326474 7609 heartbeater.cc:503] Master 127.0.98.60:45213 was elected leader, sending a full tablet report...
I20260927 12:46:40.326892 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:40.326660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40735 (local address 127.0.98.2:57895)
0927 12:46:40.326788 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:40.326834 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260927 12:46:40.327240 7745 heartbeater.cc:503] Master 127.0.98.60:45213 was elected leader, sending a full tablet report...
I20260927 12:46:40.327601 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:40.327335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40735 (local address 127.0.98.1:47981)
0927 12:46:40.327483 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:40.327534 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260927 12:46:40.327981 7612 tablet.cc:2369] T 3fe884d6b720451b952e0ae2082fa39d P 5752b93fbf53475cb5cf0593fd908fed: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 12:46:40.520870 7748 tablet.cc:2369] T 3fe884d6b720451b952e0ae2082fa39d P ae19c5cb5e6d4abc8d5bf1463c817642: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:46:40.575990 7852 raft_consensus.cc:482] T 3fe884d6b720451b952e0ae2082fa39d P 9387b2fa57714be0aa7b8e69be342b8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:46:40.576154 7852 raft_consensus.cc:504] T 3fe884d6b720451b952e0ae2082fa39d P 9387b2fa57714be0aa7b8e69be342b8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387b2fa57714be0aa7b8e69be342b8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43975 } } peers { permanent_uuid: "5752b93fbf53475cb5cf0593fd908fed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41427 } } peers { permanent_uuid: "ae19c5cb5e6d4abc8d5bf1463c817642" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45825 } }
I20260927 12:46:40.577543 7852 leader_election.cc:288] T 3fe884d6b720451b952e0ae2082fa39d P 9387b2fa57714be0aa7b8e69be342b8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5752b93fbf53475cb5cf0593fd908fed (127.0.98.2:41427), ae19c5cb5e6d4abc8d5bf1463c817642 (127.0.98.3:45825)
I20260927 12:46:40.581928 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:40.577963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33537 (local address 127.0.98.2:41427)
0927 12:46:40.578112 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:40.578116 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:40.578422 (+ 306us) server_negotiation.cc:407] Connection header received
0927 12:46:40.578485 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:40.578487 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:40.578531 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:40.578592 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:46:40.579001 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:40.579791 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:40.581003 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:40.581409 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:40.581502 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:40.581564 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:40.581737 (+ 173us) server_negotiation.cc:299] Negotiation successful
0927 12:46:40.581816 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 12:46:40.581928 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:40.577904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41427 (local address 127.0.98.1:33537)
0927 12:46:40.578215 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:40.578230 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:40.578344 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:40.578721 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:40.578724 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:40.578745 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:46:40.578875 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:40.578881 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:40.579929 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:40.579932 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:40.580855 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:40.580866 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:40.581524 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:40.581527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:40.581621 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:40.581639 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 12:46:40.581713 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 12:46:40.581787 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"mutex_wait_us":29}
I20260927 12:46:40.582581 7563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe884d6b720451b952e0ae2082fa39d" candidate_uuid: "9387b2fa57714be0aa7b8e69be342b8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5752b93fbf53475cb5cf0593fd908fed" is_pre_election: true
I20260927 12:46:40.582839 7563 raft_consensus.cc:2454] T 3fe884d6b720451b952e0ae2082fa39d P 5752b93fbf53475cb5cf0593fd908fed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9387b2fa57714be0aa7b8e69be342b8e in term 0.
I20260927 12:46:40.583405 7360 leader_election.cc:302] T 3fe884d6b720451b952e0ae2082fa39d P 9387b2fa57714be0aa7b8e69be342b8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5752b93fbf53475cb5cf0593fd908fed, 9387b2fa57714be0aa7b8e69be342b8e; no voters:
I20260927 12:46:40.583640 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:40.578928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50651 (local address 127.0.98.3:45825)
0927 12:46:40.579058 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:40.579064 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:40.579355 (+ 291us) server_negotiation.cc:407] Connection header received
0927 12:46:40.579553 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:40.579557 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:40.579612 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:40.579664 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:46:40.580105 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:40.580882 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:40.582464 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:40.583065 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:40.583186 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:40.583273 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:40.583445 (+ 172us) server_negotiation.cc:299] Negotiation successful
0927 12:46:40.583527 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 12:46:40.583729 7852 raft_consensus.cc:2781] T 3fe884d6b720451b952e0ae2082fa39d P 9387b2fa57714be0aa7b8e69be342b8e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:46:40.583782 7852 raft_consensus.cc:482] T 3fe884d6b720451b952e0ae2082fa39d P 9387b2fa57714be0aa7b8e69be342b8e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:46:40.583819 7852 raft_consensus.cc:3037] T 3fe884d6b720451b952e0ae2082fa39d P 9387b2fa57714be0aa7b8e69be342b8e [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:46:40.583968 7856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:40.578442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45825 (local address 127.0.98.1:50651)
0927 12:46:40.579262 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:40.579278 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:40.579428 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:40.579811 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:40.579814 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:40.579831 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:46:40.579974 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:40.579981 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:40.581052 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:40.581055 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:40.582321 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:40.582333 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:40.583181 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:40.583186 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:40.583303 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:40.583335 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 12:46:40.583424 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 12:46:40.583500 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":752,"mutex_wait_us":3,"thread_start_us":147,"threads_started":1}
I20260927 12:46:40.584685 7699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe884d6b720451b952e0ae2082fa39d" candidate_uuid: "9387b2fa57714be0aa7b8e69be342b8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae19c5cb5e6d4abc8d5bf1463c817642" is_pre_election: true
I20260927 12:46:40.584858 7699 raft_consensus.cc:2454] T 3fe884d6b720451b952e0ae2082fa39d P ae19c5cb5e6d4abc8d5bf1463c817642 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9387b2fa57714be0aa7b8e69be342b8e in term 0.
I20260927 12:46:40.585013 7852 raft_consensus.cc:504] T 3fe884d6b720451b952e0ae2082fa39d P 9387b2fa57714be0aa7b8e69be342b8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387b2fa57714be0aa7b8e69be342b8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43975 } } peers { permanent_uuid: "5752b93fbf53475cb5cf0593fd908fed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41427 } } peers { permanent_uuid: "ae19c5cb5e6d4abc8d5bf1463c817642" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45825 } }
I20260927 12:46:40.585356 7852 leader_election.cc:288] T 3fe884d6b720451b952e0ae2082fa39d P 9387b2fa57714be0aa7b8e69be342b8e [CANDIDATE]: Term 1 election: Requested vote from peers 5752b93fbf53475cb5cf0593fd908fed (127.0.98.2:41427), ae19c5cb5e6d4abc8d5bf1463c817642 (127.0.98.3:45825)
I20260927 12:46:40.585765 7563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe884d6b720451b952e0ae2082fa39d" candidate_uuid: "9387b2fa57714be0aa7b8e69be342b8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5752b93fbf53475cb5cf0593fd908fed"
I20260927 12:46:40.585892 7563 raft_consensus.cc:3037] T 3fe884d6b720451b952e0ae2082fa39d P 5752b93fbf53475cb5cf0593fd908fed [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:46:40.585939 7699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe884d6b720451b952e0ae2082fa39d" candidate_uuid: "9387b2fa57714be0aa7b8e69be342b8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae19c5cb5e6d4abc8d5bf1463c817642"
I20260927 12:46:40.586095 7699 raft_consensus.cc:3037] T 3fe884d6b720451b952e0ae2082fa39d P ae19c5cb5e6d4abc8d5bf1463c817642 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:46:40.586889 7563 raft_consensus.cc:2454] T 3fe884d6b720451b952e0ae2082fa39d P 5752b93fbf53475cb5cf0593fd908fed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9387b2fa57714be0aa7b8e69be342b8e in term 1.
I20260927 12:46:40.587152 7360 leader_election.cc:302] T 3fe884d6b720451b952e0ae2082fa39d P 9387b2fa57714be0aa7b8e69be342b8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5752b93fbf53475cb5cf0593fd908fed, 9387b2fa57714be0aa7b8e69be342b8e; no voters:
I20260927 12:46:40.587327 7852 raft_consensus.cc:2781] T 3fe884d6b720451b952e0ae2082fa39d P 9387b2fa57714be0aa7b8e69be342b8e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:46:40.587333 7699 raft_consensus.cc:2454] T 3fe884d6b720451b952e0ae2082fa39d P ae19c5cb5e6d4abc8d5bf1463c817642 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9387b2fa57714be0aa7b8e69be342b8e in term 1.
I20260927 12:46:40.587580 7852 raft_consensus.cc:686] T 3fe884d6b720451b952e0ae2082fa39d P 9387b2fa57714be0aa7b8e69be342b8e [term 1 LEADER]: Becoming Leader. State: Replica: 9387b2fa57714be0aa7b8e69be342b8e, State: Running, Role: LEADER
I20260927 12:46:40.587862 7852 consensus_queue.cc:237] T 3fe884d6b720451b952e0ae2082fa39d P 9387b2fa57714be0aa7b8e69be342b8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387b2fa57714be0aa7b8e69be342b8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43975 } } peers { permanent_uuid: "5752b93fbf53475cb5cf0593fd908fed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41427 } } peers { permanent_uuid: "ae19c5cb5e6d4abc8d5bf1463c817642" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45825 } }
I20260927 12:46:40.588733 7856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:40.588441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40735 (local address 127.0.98.1:59613)
0927 12:46:40.588576 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:40.588645 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 12:46:40.591184 7287 catalog_manager.cc:5495] T 3fe884d6b720451b952e0ae2082fa39d P 9387b2fa57714be0aa7b8e69be342b8e reported cstate change: term changed from 0 to 1, leader changed from <none> to 9387b2fa57714be0aa7b8e69be342b8e (127.0.98.1). New cstate: current_term: 1 leader_uuid: "9387b2fa57714be0aa7b8e69be342b8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9387b2fa57714be0aa7b8e69be342b8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5752b93fbf53475cb5cf0593fd908fed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae19c5cb5e6d4abc8d5bf1463c817642" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45825 } health_report { overall_health: UNKNOWN } } }
I20260927 12:46:40.607738 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:40.604226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39326 (local address 127.0.98.1:43975)
0927 12:46:40.604344 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:40.604349 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:40.604362 (+ 13us) server_negotiation.cc:407] Connection header received
0927 12:46:40.604519 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:40.604522 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:40.604582 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:40.604678 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:46:40.604965 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:40.605478 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:40.606325 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:40.606536 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:40.606623 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:40.606871 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:46:40.607359 (+ 488us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:46:40.607468 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:40.607536 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 12:46:40.607592 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 12:46:40.610973 7699 raft_consensus.cc:1261] T 3fe884d6b720451b952e0ae2082fa39d P ae19c5cb5e6d4abc8d5bf1463c817642 [term 1 FOLLOWER]: Refusing update from remote peer 9387b2fa57714be0aa7b8e69be342b8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:46:40.611123 7563 raft_consensus.cc:1261] T 3fe884d6b720451b952e0ae2082fa39d P 5752b93fbf53475cb5cf0593fd908fed [term 1 FOLLOWER]: Refusing update from remote peer 9387b2fa57714be0aa7b8e69be342b8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:46:40.611670 7857 consensus_queue.cc:1035] T 3fe884d6b720451b952e0ae2082fa39d P 9387b2fa57714be0aa7b8e69be342b8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5752b93fbf53475cb5cf0593fd908fed" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:46:40.611930 7852 consensus_queue.cc:1035] T 3fe884d6b720451b952e0ae2082fa39d P 9387b2fa57714be0aa7b8e69be342b8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae19c5cb5e6d4abc8d5bf1463c817642" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:46:40.613828 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:40.613505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40735 (local address 127.0.98.3:35733)
0927 12:46:40.613667 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:40.613744 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260927 12:46:40.615487 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:40.614588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40735 (local address 127.0.98.2:59329)
0927 12:46:40.615318 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:40.615389 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":641}
I20260927 12:46:40.617517 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:40.617073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40735 (local address 127.0.98.3:42969)
0927 12:46:40.617382 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:40.617438 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238}
I20260927 12:46:40.622354 7862 mvcc.cc:205] Tried to move back new op lower bound from 7333942069696806912 to 7333942069611307008. Current Snapshot: MvccSnapshot[applied={T|T < 7333942069696806912}]
I20260927 12:46:40.622756 7861 mvcc.cc:205] Tried to move back new op lower bound from 7333942069696806912 to 7333942069611307008. Current Snapshot: MvccSnapshot[applied={T|T < 7333942069696806912}]
I20260927 12:46:40.623631 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:40.623012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40735 (local address 127.0.98.2:32897)
0927 12:46:40.623472 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:40.623538 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378}
I20260927 12:46:40.625712 7856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:40.616961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40735 (local address 127.0.98.1:41321)
0927 12:46:40.625549 (+ 8588us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:40.625622 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":8499}
I20260927 12:46:40.626461 7856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:40.626218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40735 (local address 127.0.98.1:43139)
0927 12:46:40.626332 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:40.626390 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 12:46:40.640213 7285 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:46:40.640419 7285 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:46:40.645226 7285 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 70b88e0990ca471d93129ebffc98d9cf: Sending DeleteTablet for 3 replicas of tablet 3fe884d6b720451b952e0ae2082fa39d
I20260927 12:46:40.646178 7406 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fe884d6b720451b952e0ae2082fa39d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:46:40 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:39314
I20260927 12:46:40.646293 7679 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fe884d6b720451b952e0ae2082fa39d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:46:40 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:45020
I20260927 12:46:40.646675 7869 tablet_replica.cc:330] T 3fe884d6b720451b952e0ae2082fa39d P 9387b2fa57714be0aa7b8e69be342b8e: stopping tablet replica
I20260927 12:46:40.646677 7543 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fe884d6b720451b952e0ae2082fa39d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:46:40 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:43024
I20260927 12:46:40.646801 7869 raft_consensus.cc:2229] T 3fe884d6b720451b952e0ae2082fa39d P 9387b2fa57714be0aa7b8e69be342b8e [term 1 LEADER]: Raft consensus shutting down.
I20260927 12:46:40.647233 7869 raft_consensus.cc:2258] T 3fe884d6b720451b952e0ae2082fa39d P 9387b2fa57714be0aa7b8e69be342b8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:46:40.647928 7869 ts_tablet_manager.cc:1905] T 3fe884d6b720451b952e0ae2082fa39d P 9387b2fa57714be0aa7b8e69be342b8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:46:40.648738 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 7254
I20260927 12:46:40.649464 7871 tablet_replica.cc:330] T 3fe884d6b720451b952e0ae2082fa39d P 5752b93fbf53475cb5cf0593fd908fed: stopping tablet replica
I20260927 12:46:40.649592 7871 raft_consensus.cc:2229] T 3fe884d6b720451b952e0ae2082fa39d P 5752b93fbf53475cb5cf0593fd908fed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:46:40.649715 7871 raft_consensus.cc:2258] T 3fe884d6b720451b952e0ae2082fa39d P 5752b93fbf53475cb5cf0593fd908fed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:46:40.651634 7870 tablet_replica.cc:330] T 3fe884d6b720451b952e0ae2082fa39d P ae19c5cb5e6d4abc8d5bf1463c817642: stopping tablet replica
I20260927 12:46:40.651787 7870 raft_consensus.cc:2229] T 3fe884d6b720451b952e0ae2082fa39d P ae19c5cb5e6d4abc8d5bf1463c817642 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:46:40.651933 7870 raft_consensus.cc:2258] T 3fe884d6b720451b952e0ae2082fa39d P ae19c5cb5e6d4abc8d5bf1463c817642 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:46:40.653010 7871 ts_tablet_manager.cc:1905] T 3fe884d6b720451b952e0ae2082fa39d P 5752b93fbf53475cb5cf0593fd908fed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:46:40.654726 7870 ts_tablet_manager.cc:1905] T 3fe884d6b720451b952e0ae2082fa39d P ae19c5cb5e6d4abc8d5bf1463c817642: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:46:40.655947 7871 ts_tablet_manager.cc:1918] T 3fe884d6b720451b952e0ae2082fa39d P 5752b93fbf53475cb5cf0593fd908fed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:46:40.656039 7871 log.cc:1196] T 3fe884d6b720451b952e0ae2082fa39d P 5752b93fbf53475cb5cf0593fd908fed: Deleting WAL directory at /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-1/wal/wals/3fe884d6b720451b952e0ae2082fa39d
I20260927 12:46:40.656315 7871 ts_tablet_manager.cc:1939] T 3fe884d6b720451b952e0ae2082fa39d P 5752b93fbf53475cb5cf0593fd908fed: Deleting consensus metadata
I20260927 12:46:40.657080 7278 catalog_manager.cc:4841] TS 5752b93fbf53475cb5cf0593fd908fed (127.0.98.2:41427): tablet 3fe884d6b720451b952e0ae2082fa39d (table test-table [id=b9c46bdbcf4b46ada905e569b2ee53a9]) successfully deleted
I20260927 12:46:40.657673 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 7480
I20260927 12:46:40.657868 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:40.657606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43975 (local address 127.0.0.1:39340)
0927 12:46:40.657722 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:40.657779 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 12:46:40.658380 7280 catalog_manager.cc:4584] TS 9387b2fa57714be0aa7b8e69be342b8e (127.0.98.1:43975): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3fe884d6b720451b952e0ae2082fa39d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43975: connect: Connection refused (error 111)
I20260927 12:46:40.659116 7870 ts_tablet_manager.cc:1918] T 3fe884d6b720451b952e0ae2082fa39d P ae19c5cb5e6d4abc8d5bf1463c817642: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:46:40.659207 7870 log.cc:1196] T 3fe884d6b720451b952e0ae2082fa39d P ae19c5cb5e6d4abc8d5bf1463c817642: Deleting WAL directory at /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790513161160093-392-0/minicluster-data/ts-2/wal/wals/3fe884d6b720451b952e0ae2082fa39d
I20260927 12:46:40.659453 7870 ts_tablet_manager.cc:1939] T 3fe884d6b720451b952e0ae2082fa39d P ae19c5cb5e6d4abc8d5bf1463c817642: Deleting consensus metadata
I20260927 12:46:40.660158 7278 catalog_manager.cc:4841] TS ae19c5cb5e6d4abc8d5bf1463c817642 (127.0.98.3:45825): tablet 3fe884d6b720451b952e0ae2082fa39d (table test-table [id=b9c46bdbcf4b46ada905e569b2ee53a9]) successfully deleted
I20260927 12:46:40.664230 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 7616
I20260927 12:46:40.670174 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 6973
I20260927 12:46:40.675267 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 7082
2026-09-27T12:46:40Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7115 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:46:40 dist-test-slave-lkn4 krb5kdc[7874](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:46:40 dist-test-slave-lkn4 krb5kdc[7874](info): set up 2 sockets
Sep 27 12:46:40 dist-test-slave-lkn4 krb5kdc[7874](info): commencing operation
krb5kdc: starting...
W20260927 12:46:41.638480 392 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/krb5kdc/test-user.keytab.
Sep 27 12:46:41 dist-test-slave-lkn4 krb5kdc[7874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:46:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:46:41Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260927 12:46:41.759629 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42721
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42721
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:46:41.869123 7887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:46:41.869423 7887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:46:41.869464 7887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:46:41.873472 7887 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:46:41.873531 7887 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 12:46:41.873548 7887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:46:41.873564 7887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:46:41.873579 7887 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:46:41.873592 7887 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:46:41.879009 7887 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42721
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:46:41.880053 7887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:46:41.881098 7887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 12:46:41.889555 7887 server_base.cc:1034] running on GCE node
W20260927 12:46:41.889497 7892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:41.889497 7895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:41.889505 7893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:41.890470 7887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:46:41.891656 7887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:46:41.892849 7887 hybrid_clock.cc:648] HybridClock initialized: now 1790513201892823 us; error 41 us; skew 500 ppm
Sep 27 12:46:41 dist-test-slave-lkn4 krb5kdc[7874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513201, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:46:41.895797 7887 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260927 12:46:41.896726 7887 webserver.cc:460] Webserver started at http://127.0.98.62:36573/ using document root <none> and password file <none>
I20260927 12:46:41.897286 7887 fs_manager.cc:359] Metadata directory not provided
I20260927 12:46:41.897351 7887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:46:41.897543 7887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:46:41.899703 7887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/master-0/data/instance:
uuid: "27371dbb6cd84fd7be84c8e671ae4f71"
format_stamp: "Formatted at 2026-09-27 12:46:41 on dist-test-slave-lkn4"
server_key: "794cff5bcddea8a5d3386833faf29d9a"
server_key_iv: "060a5b971437950efc0013fa3292e205"
server_key_version: "encryptionkey@0"
I20260927 12:46:41.900301 7887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/master-0/wal/instance:
uuid: "27371dbb6cd84fd7be84c8e671ae4f71"
format_stamp: "Formatted at 2026-09-27 12:46:41 on dist-test-slave-lkn4"
server_key: "794cff5bcddea8a5d3386833faf29d9a"
server_key_iv: "060a5b971437950efc0013fa3292e205"
server_key_version: "encryptionkey@0"
I20260927 12:46:41.904189 7887 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260927 12:46:41.907044 7902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:41.908352 7887 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:46:41.908486 7887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/master-0/wal
uuid: "27371dbb6cd84fd7be84c8e671ae4f71"
format_stamp: "Formatted at 2026-09-27 12:46:41 on dist-test-slave-lkn4"
server_key: "794cff5bcddea8a5d3386833faf29d9a"
server_key_iv: "060a5b971437950efc0013fa3292e205"
server_key_version: "encryptionkey@0"
I20260927 12:46:41.908591 7887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:46:41.928362 7887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:46:41.931565 7887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:46:41.931746 7887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:46:41.939836 7887 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42721
I20260927 12:46:41.939934 7964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42721 every 8 connection(s)
I20260927 12:46:41.940884 7887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:46:41.943972 7965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:46:41.946146 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 7887
I20260927 12:46:41.946260 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/master-0/wal/instance
I20260927 12:46:41.946646 392 external_mini_cluster.cc:1428] Setting key 5366d571e7f4828ff9124219d0d8b7b0
I20260927 12:46:41.950150 7965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27371dbb6cd84fd7be84c8e671ae4f71: Bootstrap starting.
I20260927 12:46:41.952899 7965 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27371dbb6cd84fd7be84c8e671ae4f71: Neither blocks nor log segments found. Creating new log.
I20260927 12:46:41.953727 7965 log.cc:824] T 00000000000000000000000000000000 P 27371dbb6cd84fd7be84c8e671ae4f71: Log is configured to *not* fsync() on all Append() calls
Sep 27 12:46:41 dist-test-slave-lkn4 krb5kdc[7874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 12:46:41.956506 7965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27371dbb6cd84fd7be84c8e671ae4f71: No bootstrap required, opened a new log
I20260927 12:46:41.959992 7965 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27371dbb6cd84fd7be84c8e671ae4f71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27371dbb6cd84fd7be84c8e671ae4f71" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42721 } }
I20260927 12:46:41.960234 7965 raft_consensus.cc:374] T 00000000000000000000000000000000 P 27371dbb6cd84fd7be84c8e671ae4f71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:46:41.960307 7965 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27371dbb6cd84fd7be84c8e671ae4f71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27371dbb6cd84fd7be84c8e671ae4f71, State: Initialized, Role: FOLLOWER
I20260927 12:46:41.960794 7965 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27371dbb6cd84fd7be84c8e671ae4f71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27371dbb6cd84fd7be84c8e671ae4f71" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42721 } }
I20260927 12:46:41.960932 7965 raft_consensus.cc:388] T 00000000000000000000000000000000 P 27371dbb6cd84fd7be84c8e671ae4f71 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:46:41.960999 7965 raft_consensus.cc:482] T 00000000000000000000000000000000 P 27371dbb6cd84fd7be84c8e671ae4f71 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:46:41.961095 7965 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27371dbb6cd84fd7be84c8e671ae4f71 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:46:41.961380 7968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:41.948168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49970 (local address 127.0.98.62:42721)
0927 12:46:41.948579 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:41.948595 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:41.948634 (+ 39us) server_negotiation.cc:407] Connection header received
0927 12:46:41.949295 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:41.949315 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:41.949673 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:41.949978 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:41.950978 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:41.952022 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:41.953185 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:41.953682 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:41.953807 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:41.956891 (+ 3084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:41.956896 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:41.956910 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:41.956940 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:41.958588 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:41.959006 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:41.959010 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:41.959015 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:41.959126 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:41.959383 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:41.959386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:41.959387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:41.959793 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:41.959986 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:41.960249 (+ 263us) server_negotiation.cc:299] Negotiation successful
0927 12:46:41.960539 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":124,"threads_started":1}
I20260927 12:46:41.962069 7965 raft_consensus.cc:504] T 00000000000000000000000000000000 P 27371dbb6cd84fd7be84c8e671ae4f71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27371dbb6cd84fd7be84c8e671ae4f71" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42721 } }
I20260927 12:46:41.962404 7965 leader_election.cc:302] T 00000000000000000000000000000000 P 27371dbb6cd84fd7be84c8e671ae4f71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 27371dbb6cd84fd7be84c8e671ae4f71; no voters:
I20260927 12:46:41.962704 7965 leader_election.cc:288] T 00000000000000000000000000000000 P 27371dbb6cd84fd7be84c8e671ae4f71 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:46:41.962791 7970 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 27371dbb6cd84fd7be84c8e671ae4f71 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:46:41.962947 7970 raft_consensus.cc:686] T 00000000000000000000000000000000 P 27371dbb6cd84fd7be84c8e671ae4f71 [term 1 LEADER]: Becoming Leader. State: Replica: 27371dbb6cd84fd7be84c8e671ae4f71, State: Running, Role: LEADER
I20260927 12:46:41.963320 7970 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27371dbb6cd84fd7be84c8e671ae4f71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27371dbb6cd84fd7be84c8e671ae4f71" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42721 } }
I20260927 12:46:41.964004 7965 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27371dbb6cd84fd7be84c8e671ae4f71 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:46:41.965023 7971 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27371dbb6cd84fd7be84c8e671ae4f71 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "27371dbb6cd84fd7be84c8e671ae4f71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27371dbb6cd84fd7be84c8e671ae4f71" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42721 } } }
I20260927 12:46:41.965049 7972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27371dbb6cd84fd7be84c8e671ae4f71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27371dbb6cd84fd7be84c8e671ae4f71. Latest consensus state: current_term: 1 leader_uuid: "27371dbb6cd84fd7be84c8e671ae4f71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27371dbb6cd84fd7be84c8e671ae4f71" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42721 } } }
I20260927 12:46:41.965140 7971 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27371dbb6cd84fd7be84c8e671ae4f71 [sys.catalog]: This master's current role is: LEADER
I20260927 12:46:41.965153 7972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27371dbb6cd84fd7be84c8e671ae4f71 [sys.catalog]: This master's current role is: LEADER
I20260927 12:46:41.965503 7979 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:46:41.968658 7979 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:46:41.974382 7979 catalog_manager.cc:1347] Generated new cluster ID: 91901f42551b4c90837325e78203bf1d
I20260927 12:46:41.974469 7979 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:46:42.006886 7979 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:46:42.007953 7979 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:46:42.019399 7979 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 27371dbb6cd84fd7be84c8e671ae4f71: Generated new TSK 0
I20260927 12:46:42.020282 7979 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260927 12:46:42.052690 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42721
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:46:42.167083 7991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:46:42.167372 7991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:46:42.167438 7991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:46:42.171607 7991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:46:42.171765 7991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:46:42.171838 7991 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 12:46:42.171895 7991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:46:42.177451 7991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:46:42.178602 7991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:46:42.179659 7991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:46:42.189437 7999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:42.189446 7997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:42.189316 7996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:42.189651 7991 server_base.cc:1034] running on GCE node
I20260927 12:46:42.190218 7991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:46:42.190848 7991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:46:42.192026 7991 hybrid_clock.cc:648] HybridClock initialized: now 1790513202192003 us; error 35 us; skew 500 ppm
Sep 27 12:46:42 dist-test-slave-lkn4 krb5kdc[7874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513202, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:46:42.194950 7991 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 12:46:42.195921 7991 webserver.cc:460] Webserver started at http://127.0.98.1:36447/ using document root <none> and password file <none>
I20260927 12:46:42.196445 7991 fs_manager.cc:359] Metadata directory not provided
I20260927 12:46:42.196506 7991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:46:42.196672 7991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:46:42.198886 7991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/ts-0/data/instance:
uuid: "7b5ece2da9d94112a22a017cf744fdd4"
format_stamp: "Formatted at 2026-09-27 12:46:42 on dist-test-slave-lkn4"
server_key: "e1b0a3045f60884611faaa690e8dc42e"
server_key_iv: "34ef885c62824c442c23c479900d5140"
server_key_version: "encryptionkey@0"
I20260927 12:46:42.199420 7991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7b5ece2da9d94112a22a017cf744fdd4"
format_stamp: "Formatted at 2026-09-27 12:46:42 on dist-test-slave-lkn4"
server_key: "e1b0a3045f60884611faaa690e8dc42e"
server_key_iv: "34ef885c62824c442c23c479900d5140"
server_key_version: "encryptionkey@0"
I20260927 12:46:42.203366 7991 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 12:46:42.205899 8006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:42.207170 7991 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:46:42.207267 7991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/ts-0/wal
uuid: "7b5ece2da9d94112a22a017cf744fdd4"
format_stamp: "Formatted at 2026-09-27 12:46:42 on dist-test-slave-lkn4"
server_key: "e1b0a3045f60884611faaa690e8dc42e"
server_key_iv: "34ef885c62824c442c23c479900d5140"
server_key_version: "encryptionkey@0"
I20260927 12:46:42.207369 7991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:46:42.238021 7991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:46:42.241161 7991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:46:42.241346 7991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:46:42.242920 7991 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:46:42.242975 7991 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:42.243014 7991 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:46:42.243028 7991 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 12:46:42 dist-test-slave-lkn4 krb5kdc[7874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513202, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 12:46:42.253024 7991 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37315
I20260927 12:46:42.253158 8121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37315 every 8 connection(s)
I20260927 12:46:42.254009 7991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:46:42.256703 7968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:42.243662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50189 (local address 127.0.98.62:42721)
0927 12:46:42.243823 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:42.243828 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:42.244722 (+ 894us) server_negotiation.cc:407] Connection header received
0927 12:46:42.246025 (+ 1303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:42.246029 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:42.246130 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:42.246236 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:42.247683 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:42.248262 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:42.249438 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:42.249701 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:42.249804 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:42.252927 (+ 3123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:42.252930 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:42.252933 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:42.252959 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:42.254863 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:42.255316 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:42.255319 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:42.255320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:42.255404 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:42.255724 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:42.255726 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:42.255727 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:42.255913 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:42.255993 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:42.256450 (+ 457us) server_negotiation.cc:299] Negotiation successful
0927 12:46:42.256559 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 12:46:42.257455 8015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:42.244048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42721 (local address 127.0.98.1:50189)
0927 12:46:42.244533 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:42.244598 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:42.245695 (+ 1097us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:42.246432 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:42.246442 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:42.246860 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:42.247518 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:42.247535 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:42.248381 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:42.248384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:42.249284 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:42.249297 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:42.249871 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:42.249876 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:42.250096 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:42.251273 (+ 1177us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:42.251290 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:42.252751 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:42.254975 (+ 2224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:42.254979 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:42.254991 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:42.255228 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:42.255476 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:42.255478 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:42.255479 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:42.255595 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:42.256000 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:42.256006 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:42.256229 (+ 223us) client_negotiation.cc:769] Sending connection context
0927 12:46:42.256380 (+ 151us) client_negotiation.cc:240] Negotiation successful
0927 12:46:42.256622 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":107,"threads_started":1}
I20260927 12:46:42.258823 8122 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42721
I20260927 12:46:42.258965 8122 heartbeater.cc:464] Registering TS with master...
I20260927 12:46:42.259495 8122 heartbeater.cc:511] Master 127.0.98.62:42721 requested a full tablet report, sending...
I20260927 12:46:42.259627 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 7991
I20260927 12:46:42.259744 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:46:42.260046 392 external_mini_cluster.cc:1428] Setting key cb9a892e754aa26c3bd0804324a7ee04
I20260927 12:46:42.261323 7918 ts_manager.cc:194] Registered new tserver with Master: 7b5ece2da9d94112a22a017cf744fdd4 (127.0.98.1:37315)
I20260927 12:46:42.262820 7918 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50189
I20260927 12:46:42.268607 7968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:42.261155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51783 (local address 127.0.98.62:42721)
0927 12:46:42.261334 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:42.261338 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:42.261352 (+ 14us) server_negotiation.cc:407] Connection header received
0927 12:46:42.261385 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:42.261388 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:42.261429 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:42.261504 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:42.261996 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:42.262793 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:42.263889 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:42.264174 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:42.264258 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:42.265554 (+ 1296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:42.265557 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:42.265559 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:42.265580 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:42.267016 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:42.267478 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:42.267481 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:42.267482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:42.267556 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:42.267845 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:42.267848 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:42.267849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:42.268027 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:42.268108 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:42.268389 (+ 281us) server_negotiation.cc:299] Negotiation successful
0927 12:46:42.268491 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260927 12:46:42.268766 8128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:42.260831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42721 (local address 127.0.98.1:51783)
0927 12:46:42.261113 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:42.261127 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:42.261207 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:42.261527 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:42.261530 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:42.261752 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:42.261879 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:42.261888 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:42.262941 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:42.262944 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:42.263728 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:42.263740 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:42.264285 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:42.264288 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:42.264433 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:42.264971 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:42.264985 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:42.265384 (+ 399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:42.267122 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:42.267126 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:42.267128 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:42.267385 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:42.267635 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:42.267638 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:42.267641 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:42.267736 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:42.268115 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:42.268119 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:42.268227 (+ 108us) client_negotiation.cc:769] Sending connection context
0927 12:46:42.268334 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 12:46:42.268480 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":101,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260927 12:46:42.286459 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42721
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:46:42.395144 8131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:46:42.395471 8131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:46:42.395521 8131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:46:42.399616 8131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:46:42.399680 8131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:46:42.399719 8131 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 12:46:42.399761 8131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:46:42.405040 8131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8131
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:46:42.406129 8131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:46:42.407182 8131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:46:42.417291 8136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:42.417351 8139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:42.417291 8137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:42.417606 8131 server_base.cc:1034] running on GCE node
I20260927 12:46:42.418130 8131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:46:42.418855 8131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:46:42.420053 8131 hybrid_clock.cc:648] HybridClock initialized: now 1790513202420018 us; error 51 us; skew 500 ppm
Sep 27 12:46:42 dist-test-slave-lkn4 krb5kdc[7874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513202, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:46:42.423130 8131 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 12:46:42.424183 8131 webserver.cc:460] Webserver started at http://127.0.98.2:40433/ using document root <none> and password file <none>
I20260927 12:46:42.424773 8131 fs_manager.cc:359] Metadata directory not provided
I20260927 12:46:42.424842 8131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:46:42.425047 8131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:46:42.427194 8131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/ts-1/data/instance:
uuid: "d016d468d37b4edbab192d8c94d556bd"
format_stamp: "Formatted at 2026-09-27 12:46:42 on dist-test-slave-lkn4"
server_key: "704db12f348d842ad89630986b09d5b5"
server_key_iv: "1b892002f54314f1dc4bdf44b572d325"
server_key_version: "encryptionkey@0"
I20260927 12:46:42.427747 8131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d016d468d37b4edbab192d8c94d556bd"
format_stamp: "Formatted at 2026-09-27 12:46:42 on dist-test-slave-lkn4"
server_key: "704db12f348d842ad89630986b09d5b5"
server_key_iv: "1b892002f54314f1dc4bdf44b572d325"
server_key_version: "encryptionkey@0"
I20260927 12:46:42.431481 8131 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 12:46:42.434093 8146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:42.435496 8131 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:46:42.435631 8131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/ts-1/wal
uuid: "d016d468d37b4edbab192d8c94d556bd"
format_stamp: "Formatted at 2026-09-27 12:46:42 on dist-test-slave-lkn4"
server_key: "704db12f348d842ad89630986b09d5b5"
server_key_iv: "1b892002f54314f1dc4bdf44b572d325"
server_key_version: "encryptionkey@0"
I20260927 12:46:42.435748 8131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:46:42.459038 8131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:46:42.462117 8131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:46:42.462286 8131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:46:42.463670 8131 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:46:42.463723 8131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:42.463785 8131 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:46:42.463810 8131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:42.473615 8131 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34063
I20260927 12:46:42.473671 8261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34063 every 8 connection(s)
Sep 27 12:46:42 dist-test-slave-lkn4 krb5kdc[7874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513202, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 12:46:42.474591 8131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:46:42.477689 7968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:42.464548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38965 (local address 127.0.98.62:42721)
0927 12:46:42.464686 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:42.464690 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:42.465552 (+ 862us) server_negotiation.cc:407] Connection header received
0927 12:46:42.466793 (+ 1241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:42.466796 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:42.466855 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:42.466941 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:42.468619 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:42.469471 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:42.471166 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:42.471385 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:42.471470 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:42.474431 (+ 2961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:42.474435 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:42.474438 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:42.474467 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:42.475818 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:42.476284 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:42.476287 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:42.476289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:42.476361 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:42.476664 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:42.476667 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:42.476669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:42.476844 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:42.476927 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:42.477405 (+ 478us) server_negotiation.cc:299] Negotiation successful
0927 12:46:42.477521 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 12:46:42.478478 8155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:42.464981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42721 (local address 127.0.98.2:38965)
0927 12:46:42.465405 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:42.465452 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:42.466524 (+ 1072us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:42.467267 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:42.467276 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:42.467736 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:42.468447 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:42.468466 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:42.469609 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:42.469613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:42.471030 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:42.471040 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:42.471539 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:42.471544 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:42.471752 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:42.472471 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:42.472492 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:42.474267 (+ 1775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:42.475920 (+ 1653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:42.475926 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:42.475940 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:42.476184 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:42.476437 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:42.476440 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:42.476443 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:42.476567 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:42.476936 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:42.476943 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:42.477180 (+ 237us) client_negotiation.cc:769] Sending connection context
0927 12:46:42.477354 (+ 174us) client_negotiation.cc:240] Negotiation successful
0927 12:46:42.477659 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":107,"threads_started":1}
I20260927 12:46:42.479609 8262 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42721
I20260927 12:46:42.479749 8262 heartbeater.cc:464] Registering TS with master...
I20260927 12:46:42.480327 8262 heartbeater.cc:511] Master 127.0.98.62:42721 requested a full tablet report, sending...
I20260927 12:46:42.481370 7919 ts_manager.cc:194] Registered new tserver with Master: d016d468d37b4edbab192d8c94d556bd (127.0.98.2:34063)
I20260927 12:46:42.482118 7919 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38965
I20260927 12:46:42.482597 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 8131
I20260927 12:46:42.482719 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:46:42.482955 392 external_mini_cluster.cc:1428] Setting key 5a679b051ea7ae00f2bc1ab24123ff9f
I20260927 12:46:42.489307 7968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:42.482019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54809 (local address 127.0.98.62:42721)
0927 12:46:42.482153 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:42.482157 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:42.482390 (+ 233us) server_negotiation.cc:407] Connection header received
0927 12:46:42.482517 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:42.482520 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:42.482555 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:42.482592 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:42.483205 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:42.483848 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:42.484713 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:42.484933 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:42.485017 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:42.486241 (+ 1224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:42.486244 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:42.486246 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:42.486267 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:42.487593 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:42.488112 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:42.488115 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:42.488116 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:42.488188 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:42.488470 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:42.488473 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:42.488474 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:42.488748 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:42.488832 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:42.489092 (+ 260us) server_negotiation.cc:299] Negotiation successful
0927 12:46:42.489184 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 12:46:42.489415 8268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:42.481921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42721 (local address 127.0.98.2:54809)
0927 12:46:42.482303 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:42.482316 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:42.482413 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:42.482697 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:42.482700 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:42.482917 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:42.483063 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:42.483071 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:42.483981 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:42.483984 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:42.484605 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:42.484614 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:42.485029 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:42.485033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:42.485118 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:42.485613 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:42.485627 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:42.486083 (+ 456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:42.487777 (+ 1694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:42.487781 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:42.487783 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:42.488027 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:42.488266 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:42.488269 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:42.488271 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:42.488363 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:42.488841 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:42.488844 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:42.488945 (+ 101us) client_negotiation.cc:769] Sending connection context
0927 12:46:42.489058 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 12:46:42.489200 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"mutex_wait_us":3,"thread_start_us":117,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 12:46:42.509352 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42721
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:46:42.619683 8271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:46:42.619974 8271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:46:42.620038 8271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:46:42.624135 8271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:46:42.624212 8271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:46:42.624276 8271 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 12:46:42.624330 8271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:46:42.629714 8271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:46:42.630820 8271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:46:42.631821 8271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 12:46:42.641510 8271 server_base.cc:1034] running on GCE node
W20260927 12:46:42.641451 8277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:42.641459 8276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:42.641460 8279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:42.642252 8271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:46:42.642903 8271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:46:42.644099 8271 hybrid_clock.cc:648] HybridClock initialized: now 1790513202644077 us; error 36 us; skew 500 ppm
Sep 27 12:46:42 dist-test-slave-lkn4 krb5kdc[7874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513202, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:46:42.646994 8271 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 12:46:42.647986 8271 webserver.cc:460] Webserver started at http://127.0.98.3:33083/ using document root <none> and password file <none>
I20260927 12:46:42.648548 8271 fs_manager.cc:359] Metadata directory not provided
I20260927 12:46:42.648624 8271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:46:42.648846 8271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:46:42.651090 8271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/ts-2/data/instance:
uuid: "655d707e276145a9a712ab8388e1d5d8"
format_stamp: "Formatted at 2026-09-27 12:46:42 on dist-test-slave-lkn4"
server_key: "72bb4a5ff32d756d0d5c4cdcecdb2168"
server_key_iv: "1b83dafd7e0488266c927fb39b93d55e"
server_key_version: "encryptionkey@0"
I20260927 12:46:42.651690 8271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/ts-2/wal/instance:
uuid: "655d707e276145a9a712ab8388e1d5d8"
format_stamp: "Formatted at 2026-09-27 12:46:42 on dist-test-slave-lkn4"
server_key: "72bb4a5ff32d756d0d5c4cdcecdb2168"
server_key_iv: "1b83dafd7e0488266c927fb39b93d55e"
server_key_version: "encryptionkey@0"
I20260927 12:46:42.655714 8271 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 12:46:42.658486 8286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:42.659960 8271 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:46:42.660091 8271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/ts-2/wal
uuid: "655d707e276145a9a712ab8388e1d5d8"
format_stamp: "Formatted at 2026-09-27 12:46:42 on dist-test-slave-lkn4"
server_key: "72bb4a5ff32d756d0d5c4cdcecdb2168"
server_key_iv: "1b83dafd7e0488266c927fb39b93d55e"
server_key_version: "encryptionkey@0"
I20260927 12:46:42.660214 8271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:46:42.693578 8271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:46:42.697093 8271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:46:42.697299 8271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:46:42.698957 8271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:46:42.699010 8271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:42.699067 8271 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:46:42.699095 8271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 12:46:42 dist-test-slave-lkn4 krb5kdc[7874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513202, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 12:46:42.709532 8271 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38265
I20260927 12:46:42.709674 8401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38265 every 8 connection(s)
I20260927 12:46:42.710731 8271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:46:42.713466 7968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:42.699854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54743 (local address 127.0.98.62:42721)
0927 12:46:42.699980 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:42.699984 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:42.701079 (+ 1095us) server_negotiation.cc:407] Connection header received
0927 12:46:42.702262 (+ 1183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:42.702267 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:42.702326 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:42.702401 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:42.703923 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:42.704715 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:42.706265 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:42.706523 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:42.706631 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:42.709574 (+ 2943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:42.709578 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:42.709581 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:42.709617 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:42.711291 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:42.711880 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:42.711883 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:42.711885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:42.712004 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:42.712326 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:42.712329 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:42.712331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:42.712540 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:42.712627 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:42.713179 (+ 552us) server_negotiation.cc:299] Negotiation successful
0927 12:46:42.713306 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 12:46:42.714371 8295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:42.700397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42721 (local address 127.0.98.3:54743)
0927 12:46:42.700914 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:42.700958 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:42.701995 (+ 1037us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:42.702732 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:42.702743 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:42.703162 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:42.703765 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:42.703783 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:42.704830 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:42.704834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:42.706140 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:42.706149 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:42.706672 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:42.706676 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:42.706882 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:42.707537 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:42.707560 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:42.709357 (+ 1797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:42.711412 (+ 2055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:42.711418 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:42.711434 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:42.711779 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:42.712097 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:42.712100 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:42.712102 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:42.712236 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:42.712654 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:42.712660 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:42.712920 (+ 260us) client_negotiation.cc:769] Sending connection context
0927 12:46:42.713130 (+ 210us) client_negotiation.cc:240] Negotiation successful
0927 12:46:42.713463 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":129,"threads_started":1}
I20260927 12:46:42.715629 8402 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42721
I20260927 12:46:42.715830 8402 heartbeater.cc:464] Registering TS with master...
I20260927 12:46:42.716212 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 8271
I20260927 12:46:42.716329 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:46:42.716447 8402 heartbeater.cc:511] Master 127.0.98.62:42721 requested a full tablet report, sending...
I20260927 12:46:42.716653 392 external_mini_cluster.cc:1428] Setting key 58916075d9075f47277666f6c6f10b42
I20260927 12:46:42.717499 7919 ts_manager.cc:194] Registered new tserver with Master: 655d707e276145a9a712ab8388e1d5d8 (127.0.98.3:38265)
I20260927 12:46:42.718185 7919 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54743
I20260927 12:46:42.719105 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 12:46:42.726207 7968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:42.718122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55555 (local address 127.0.98.62:42721)
0927 12:46:42.718239 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:42.718244 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:42.718583 (+ 339us) server_negotiation.cc:407] Connection header received
0927 12:46:42.718770 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:42.718773 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:42.718823 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:42.718901 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:42.719503 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:42.720222 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:42.721205 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:42.721431 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:42.721529 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:42.722775 (+ 1246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:42.722779 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:42.722782 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:42.722808 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:42.724461 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:42.724992 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:42.724995 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:42.724997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:42.725094 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:42.725399 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:42.725401 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:42.725402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:42.725572 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:42.725643 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:42.725947 (+ 304us) server_negotiation.cc:299] Negotiation successful
0927 12:46:42.726081 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 12:46:42.726308 8408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:42.718026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42721 (local address 127.0.98.3:55555)
0927 12:46:42.718507 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:42.718522 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:42.718658 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:42.719006 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:42.719009 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:42.719228 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:42.719342 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:42.719349 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:42.720362 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:42.720365 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:42.721050 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:42.721060 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:42.721538 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:42.721543 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:42.721647 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:42.722211 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:42.722224 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:42.722636 (+ 412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:42.724599 (+ 1963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:42.724602 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:42.724603 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:42.724892 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:42.725197 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:42.725200 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:42.725202 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:42.725308 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:42.725655 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:42.725659 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:42.725768 (+ 109us) client_negotiation.cc:769] Sending connection context
0927 12:46:42.725900 (+ 132us) client_negotiation.cc:240] Negotiation successful
0927 12:46:42.726098 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":97,"threads_started":1}
Sep 27 12:46:42 dist-test-slave-lkn4 krb5kdc[7874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513202, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 12:46:42 dist-test-slave-lkn4 krb5kdc[7874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:46:42 dist-test-slave-lkn4 krb5kdc[7874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 12:46:42.743199 7968 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:46:42.737016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49976 (local address 127.0.98.62:42721)
0927 12:46:42.737148 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:42.737152 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:42.737164 (+ 12us) server_negotiation.cc:407] Connection header received
0927 12:46:42.737198 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:42.737200 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:42.737235 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:42.737304 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:42.737978 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:42.738556 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:42.739440 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:42.739653 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:42.739740 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:42.743050 (+ 3310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49976: BlockingRecv error: recv got EOF from 127.0.0.1:49976 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 27 12:46:42 dist-test-slave-lkn4 krb5kdc[7874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513202, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 12:46:42.752216 7968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:42.744589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49978 (local address 127.0.98.62:42721)
0927 12:46:42.744700 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:42.744704 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:42.744832 (+ 128us) server_negotiation.cc:407] Connection header received
0927 12:46:42.744941 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:42.744943 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:42.744980 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:42.745045 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:42.745604 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:42.746168 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:42.746954 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:42.747162 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:42.747244 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:42.749509 (+ 2265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:42.749512 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:42.749515 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:42.749538 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:42.750843 (+ 1305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:42.751192 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:42.751194 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:42.751196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:42.751264 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:42.751504 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:42.751506 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:42.751507 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:42.751664 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:42.751802 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:42.751997 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 12:46:42.752094 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 12:46:42.755216 7918 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49978:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:46:42.758141 7918 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:46:42.772604 8424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:42.766887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34063 (local address 127.0.0.1:53392)
0927 12:46:42.767443 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:42.767493 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:42.767643 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:42.768514 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:42.768517 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:42.768536 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:46:42.768650 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:42.768657 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:42.770079 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:42.770083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:42.771159 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:42.771169 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:42.772114 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:42.772117 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:42.772221 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:42.772239 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 12:46:42.772299 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 12:46:42.772450 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"mutex_wait_us":25,"thread_start_us":91,"threads_started":1}
I20260927 12:46:42.772624 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:42.767530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53392 (local address 127.0.98.2:34063)
0927 12:46:42.768104 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:42.768115 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:42.768134 (+ 19us) server_negotiation.cc:407] Connection header received
0927 12:46:42.768198 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:42.768203 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:42.768382 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:42.768515 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:46:42.768953 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:42.769934 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:42.771297 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:42.772009 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:42.772133 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:42.772298 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:42.772352 (+ 54us) server_negotiation.cc:299] Negotiation successful
0927 12:46:42.772462 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":107,"threads_started":1}
I20260927 12:46:42.772646 8425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:42.766818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49210 (local address 127.0.98.1:37315)
0927 12:46:42.767202 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:42.767210 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:42.767576 (+ 366us) server_negotiation.cc:407] Connection header received
0927 12:46:42.767810 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:42.767818 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:42.768029 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:42.768153 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:46:42.768623 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:42.769572 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:42.770927 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:42.771744 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:42.771818 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:42.771968 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:42.772432 (+ 464us) server_negotiation.cc:299] Negotiation successful
0927 12:46:42.772528 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":84,"threads_started":1}
I20260927 12:46:42.773196 8427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:42.766710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37315 (local address 127.0.0.1:49210)
0927 12:46:42.767443 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:42.767491 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:42.767649 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:42.768373 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:42.768379 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:42.768400 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:46:42.768521 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:42.768529 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:42.769676 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:42.769679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:42.770800 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:42.770808 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:42.772046 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:42.772048 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:42.772147 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:42.772206 (+ 59us) client_negotiation.cc:769] Sending connection context
0927 12:46:42.772303 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 12:46:42.773106 (+ 803us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"mutex_wait_us":717,"thread_start_us":96,"threads_started":1}
I20260927 12:46:42.773416 8426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:42.767126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38265 (local address 127.0.0.1:45966)
0927 12:46:42.768674 (+ 1548us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:42.768684 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:42.768760 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:42.769276 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:42.769278 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:42.769294 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:46:42.769390 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:42.769395 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:42.770735 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:42.770739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:42.771880 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:42.771889 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:42.773056 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:42.773060 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:42.773151 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:42.773176 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 12:46:42.773246 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 12:46:42.773317 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1496,"mutex_wait_us":31,"thread_start_us":86,"threads_started":1}
I20260927 12:46:42.774470 8428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:42.767518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45966 (local address 127.0.98.3:38265)
0927 12:46:42.767926 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:42.767937 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:42.768764 (+ 827us) server_negotiation.cc:407] Connection header received
0927 12:46:42.768845 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:42.768850 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:42.769021 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:42.769151 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:46:42.769511 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:42.770487 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:42.771986 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:42.772754 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:42.772865 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:42.772997 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:42.773627 (+ 630us) server_negotiation.cc:299] Negotiation successful
0927 12:46:42.773724 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":110,"threads_started":1}
I20260927 12:46:42.776036 8196 tablet_service.cc:1467] Processing CreateTablet for tablet fbf9a37cf51745948a2a7df6d52f094a (DEFAULT_TABLE table=test-table [id=d16653f03bca48f992c01911d7931c92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:46:42.776033 8056 tablet_service.cc:1467] Processing CreateTablet for tablet fbf9a37cf51745948a2a7df6d52f094a (DEFAULT_TABLE table=test-table [id=d16653f03bca48f992c01911d7931c92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:46:42.776589 8336 tablet_service.cc:1467] Processing CreateTablet for tablet fbf9a37cf51745948a2a7df6d52f094a (DEFAULT_TABLE table=test-table [id=d16653f03bca48f992c01911d7931c92]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:46:42.777099 8056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbf9a37cf51745948a2a7df6d52f094a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:46:42.777100 8196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbf9a37cf51745948a2a7df6d52f094a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:46:42.777383 8336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbf9a37cf51745948a2a7df6d52f094a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:46:42.784165 8430 tablet_bootstrap.cc:492] T fbf9a37cf51745948a2a7df6d52f094a P 655d707e276145a9a712ab8388e1d5d8: Bootstrap starting.
I20260927 12:46:42.785147 8431 tablet_bootstrap.cc:492] T fbf9a37cf51745948a2a7df6d52f094a P 7b5ece2da9d94112a22a017cf744fdd4: Bootstrap starting.
I20260927 12:46:42.786430 8432 tablet_bootstrap.cc:492] T fbf9a37cf51745948a2a7df6d52f094a P d016d468d37b4edbab192d8c94d556bd: Bootstrap starting.
I20260927 12:46:42.786844 8430 tablet_bootstrap.cc:654] T fbf9a37cf51745948a2a7df6d52f094a P 655d707e276145a9a712ab8388e1d5d8: Neither blocks nor log segments found. Creating new log.
I20260927 12:46:42.787465 8431 tablet_bootstrap.cc:654] T fbf9a37cf51745948a2a7df6d52f094a P 7b5ece2da9d94112a22a017cf744fdd4: Neither blocks nor log segments found. Creating new log.
I20260927 12:46:42.787765 8430 log.cc:824] T fbf9a37cf51745948a2a7df6d52f094a P 655d707e276145a9a712ab8388e1d5d8: Log is configured to *not* fsync() on all Append() calls
I20260927 12:46:42.788245 8432 tablet_bootstrap.cc:654] T fbf9a37cf51745948a2a7df6d52f094a P d016d468d37b4edbab192d8c94d556bd: Neither blocks nor log segments found. Creating new log.
I20260927 12:46:42.788326 8431 log.cc:824] T fbf9a37cf51745948a2a7df6d52f094a P 7b5ece2da9d94112a22a017cf744fdd4: Log is configured to *not* fsync() on all Append() calls
I20260927 12:46:42.789014 8432 log.cc:824] T fbf9a37cf51745948a2a7df6d52f094a P d016d468d37b4edbab192d8c94d556bd: Log is configured to *not* fsync() on all Append() calls
I20260927 12:46:42.790004 8430 tablet_bootstrap.cc:492] T fbf9a37cf51745948a2a7df6d52f094a P 655d707e276145a9a712ab8388e1d5d8: No bootstrap required, opened a new log
I20260927 12:46:42.790225 8430 ts_tablet_manager.cc:1397] T fbf9a37cf51745948a2a7df6d52f094a P 655d707e276145a9a712ab8388e1d5d8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.002s
I20260927 12:46:42.790373 8431 tablet_bootstrap.cc:492] T fbf9a37cf51745948a2a7df6d52f094a P 7b5ece2da9d94112a22a017cf744fdd4: No bootstrap required, opened a new log
I20260927 12:46:42.790558 8431 ts_tablet_manager.cc:1397] T fbf9a37cf51745948a2a7df6d52f094a P 7b5ece2da9d94112a22a017cf744fdd4: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 12:46:42.791270 8432 tablet_bootstrap.cc:492] T fbf9a37cf51745948a2a7df6d52f094a P d016d468d37b4edbab192d8c94d556bd: No bootstrap required, opened a new log
I20260927 12:46:42.791443 8432 ts_tablet_manager.cc:1397] T fbf9a37cf51745948a2a7df6d52f094a P d016d468d37b4edbab192d8c94d556bd: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 12:46:42.794014 8431 raft_consensus.cc:348] T fbf9a37cf51745948a2a7df6d52f094a P 7b5ece2da9d94112a22a017cf744fdd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5ece2da9d94112a22a017cf744fdd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37315 } } peers { permanent_uuid: "d016d468d37b4edbab192d8c94d556bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34063 } } peers { permanent_uuid: "655d707e276145a9a712ab8388e1d5d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38265 } }
I20260927 12:46:42.794342 8431 raft_consensus.cc:374] T fbf9a37cf51745948a2a7df6d52f094a P 7b5ece2da9d94112a22a017cf744fdd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:46:42.794409 8431 raft_consensus.cc:729] T fbf9a37cf51745948a2a7df6d52f094a P 7b5ece2da9d94112a22a017cf744fdd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b5ece2da9d94112a22a017cf744fdd4, State: Initialized, Role: FOLLOWER
I20260927 12:46:42.794230 8430 raft_consensus.cc:348] T fbf9a37cf51745948a2a7df6d52f094a P 655d707e276145a9a712ab8388e1d5d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5ece2da9d94112a22a017cf744fdd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37315 } } peers { permanent_uuid: "d016d468d37b4edbab192d8c94d556bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34063 } } peers { permanent_uuid: "655d707e276145a9a712ab8388e1d5d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38265 } }
I20260927 12:46:42.794477 8430 raft_consensus.cc:374] T fbf9a37cf51745948a2a7df6d52f094a P 655d707e276145a9a712ab8388e1d5d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:46:42.794529 8430 raft_consensus.cc:729] T fbf9a37cf51745948a2a7df6d52f094a P 655d707e276145a9a712ab8388e1d5d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 655d707e276145a9a712ab8388e1d5d8, State: Initialized, Role: FOLLOWER
I20260927 12:46:42.794961 8431 consensus_queue.cc:260] T fbf9a37cf51745948a2a7df6d52f094a P 7b5ece2da9d94112a22a017cf744fdd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5ece2da9d94112a22a017cf744fdd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37315 } } peers { permanent_uuid: "d016d468d37b4edbab192d8c94d556bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34063 } } peers { permanent_uuid: "655d707e276145a9a712ab8388e1d5d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38265 } }
I20260927 12:46:42.794970 8430 consensus_queue.cc:260] T fbf9a37cf51745948a2a7df6d52f094a P 655d707e276145a9a712ab8388e1d5d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5ece2da9d94112a22a017cf744fdd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37315 } } peers { permanent_uuid: "d016d468d37b4edbab192d8c94d556bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34063 } } peers { permanent_uuid: "655d707e276145a9a712ab8388e1d5d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38265 } }
I20260927 12:46:42.795136 8432 raft_consensus.cc:348] T fbf9a37cf51745948a2a7df6d52f094a P d016d468d37b4edbab192d8c94d556bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5ece2da9d94112a22a017cf744fdd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37315 } } peers { permanent_uuid: "d016d468d37b4edbab192d8c94d556bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34063 } } peers { permanent_uuid: "655d707e276145a9a712ab8388e1d5d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38265 } }
I20260927 12:46:42.795392 8432 raft_consensus.cc:374] T fbf9a37cf51745948a2a7df6d52f094a P d016d468d37b4edbab192d8c94d556bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:46:42.795446 8432 raft_consensus.cc:729] T fbf9a37cf51745948a2a7df6d52f094a P d016d468d37b4edbab192d8c94d556bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d016d468d37b4edbab192d8c94d556bd, State: Initialized, Role: FOLLOWER
I20260927 12:46:42.795754 8122 heartbeater.cc:503] Master 127.0.98.62:42721 was elected leader, sending a full tablet report...
I20260927 12:46:42.795828 8432 consensus_queue.cc:260] T fbf9a37cf51745948a2a7df6d52f094a P d016d468d37b4edbab192d8c94d556bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5ece2da9d94112a22a017cf744fdd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37315 } } peers { permanent_uuid: "d016d468d37b4edbab192d8c94d556bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34063 } } peers { permanent_uuid: "655d707e276145a9a712ab8388e1d5d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38265 } }
I20260927 12:46:42.796015 8431 ts_tablet_manager.cc:1428] T fbf9a37cf51745948a2a7df6d52f094a P 7b5ece2da9d94112a22a017cf744fdd4: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260927 12:46:42.796023 8430 ts_tablet_manager.cc:1428] T fbf9a37cf51745948a2a7df6d52f094a P 655d707e276145a9a712ab8388e1d5d8: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260927 12:46:42.796365 8402 heartbeater.cc:503] Master 127.0.98.62:42721 was elected leader, sending a full tablet report...
I20260927 12:46:42.796595 8432 ts_tablet_manager.cc:1428] T fbf9a37cf51745948a2a7df6d52f094a P d016d468d37b4edbab192d8c94d556bd: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 12:46:42.796890 8262 heartbeater.cc:503] Master 127.0.98.62:42721 was elected leader, sending a full tablet report...
W20260927 12:46:42.961891 8403 tablet.cc:2369] T fbf9a37cf51745948a2a7df6d52f094a P 655d707e276145a9a712ab8388e1d5d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 12:46:42.976050 8263 tablet.cc:2369] T fbf9a37cf51745948a2a7df6d52f094a P d016d468d37b4edbab192d8c94d556bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 12:46:43.005482 8123 tablet.cc:2369] T fbf9a37cf51745948a2a7df6d52f094a P 7b5ece2da9d94112a22a017cf744fdd4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:46:43.005690 8437 raft_consensus.cc:482] T fbf9a37cf51745948a2a7df6d52f094a P 655d707e276145a9a712ab8388e1d5d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:46:43.005939 8437 raft_consensus.cc:504] T fbf9a37cf51745948a2a7df6d52f094a P 655d707e276145a9a712ab8388e1d5d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5ece2da9d94112a22a017cf744fdd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37315 } } peers { permanent_uuid: "d016d468d37b4edbab192d8c94d556bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34063 } } peers { permanent_uuid: "655d707e276145a9a712ab8388e1d5d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38265 } }
I20260927 12:46:43.007151 8437 leader_election.cc:288] T fbf9a37cf51745948a2a7df6d52f094a P 655d707e276145a9a712ab8388e1d5d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b5ece2da9d94112a22a017cf744fdd4 (127.0.98.1:37315), d016d468d37b4edbab192d8c94d556bd (127.0.98.2:34063)
I20260927 12:46:43.010886 8295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:43.007405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37315 (local address 127.0.98.3:39783)
0927 12:46:43.007602 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:43.007616 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:43.007726 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:43.007977 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:43.007980 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:43.007994 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:46:43.008111 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:43.008116 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:43.008819 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:43.008821 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:43.009795 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:43.009806 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:43.010482 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:43.010485 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:43.010562 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:43.010584 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 12:46:43.010652 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 12:46:43.010745 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":1}
I20260927 12:46:43.010886 8425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:43.007484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39783 (local address 127.0.98.1:37315)
0927 12:46:43.007604 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:43.007608 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:43.007682 (+ 74us) server_negotiation.cc:407] Connection header received
0927 12:46:43.007840 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:43.007843 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:43.007888 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:43.007960 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:46:43.008214 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:43.008716 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:43.009919 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:43.010375 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:43.010472 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:43.010538 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:43.010691 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 12:46:43.010773 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 12:46:43.011497 8439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:43.007452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34063 (local address 127.0.98.3:35119)
0927 12:46:43.007879 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:43.007890 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:43.007976 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:43.008183 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:43.008185 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:43.008202 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:46:43.008666 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:43.008675 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:43.009375 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:43.009378 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:43.010337 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:43.010346 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:43.011142 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:43.011146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:43.011223 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:43.011241 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 12:46:43.011307 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 12:46:43.011367 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":158,"threads_started":1}
I20260927 12:46:43.011521 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:43.007550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35119 (local address 127.0.98.2:34063)
0927 12:46:43.007764 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:43.007768 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:43.007964 (+ 196us) server_negotiation.cc:407] Connection header received
0927 12:46:43.008056 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:43.008059 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:43.008102 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:43.008161 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:46:43.008786 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:43.009275 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:43.010451 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:43.010947 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:43.011051 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:43.011110 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:43.011348 (+ 238us) server_negotiation.cc:299] Negotiation successful
0927 12:46:43.011424 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 12:46:43.011615 8076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbf9a37cf51745948a2a7df6d52f094a" candidate_uuid: "655d707e276145a9a712ab8388e1d5d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b5ece2da9d94112a22a017cf744fdd4" is_pre_election: true
I20260927 12:46:43.011973 8076 raft_consensus.cc:2454] T fbf9a37cf51745948a2a7df6d52f094a P 7b5ece2da9d94112a22a017cf744fdd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 655d707e276145a9a712ab8388e1d5d8 in term 0.
I20260927 12:46:43.012171 8215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbf9a37cf51745948a2a7df6d52f094a" candidate_uuid: "655d707e276145a9a712ab8388e1d5d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d016d468d37b4edbab192d8c94d556bd" is_pre_election: true
I20260927 12:46:43.012446 8215 raft_consensus.cc:2454] T fbf9a37cf51745948a2a7df6d52f094a P d016d468d37b4edbab192d8c94d556bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 655d707e276145a9a712ab8388e1d5d8 in term 0.
I20260927 12:46:43.012483 8289 leader_election.cc:302] T fbf9a37cf51745948a2a7df6d52f094a P 655d707e276145a9a712ab8388e1d5d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 655d707e276145a9a712ab8388e1d5d8, 7b5ece2da9d94112a22a017cf744fdd4; no voters:
I20260927 12:46:43.012766 8437 raft_consensus.cc:2781] T fbf9a37cf51745948a2a7df6d52f094a P 655d707e276145a9a712ab8388e1d5d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:46:43.012845 8437 raft_consensus.cc:482] T fbf9a37cf51745948a2a7df6d52f094a P 655d707e276145a9a712ab8388e1d5d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:46:43.012897 8437 raft_consensus.cc:3037] T fbf9a37cf51745948a2a7df6d52f094a P 655d707e276145a9a712ab8388e1d5d8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:46:43.013964 8437 raft_consensus.cc:504] T fbf9a37cf51745948a2a7df6d52f094a P 655d707e276145a9a712ab8388e1d5d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5ece2da9d94112a22a017cf744fdd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37315 } } peers { permanent_uuid: "d016d468d37b4edbab192d8c94d556bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34063 } } peers { permanent_uuid: "655d707e276145a9a712ab8388e1d5d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38265 } }
I20260927 12:46:43.014353 8437 leader_election.cc:288] T fbf9a37cf51745948a2a7df6d52f094a P 655d707e276145a9a712ab8388e1d5d8 [CANDIDATE]: Term 1 election: Requested vote from peers 7b5ece2da9d94112a22a017cf744fdd4 (127.0.98.1:37315), d016d468d37b4edbab192d8c94d556bd (127.0.98.2:34063)
I20260927 12:46:43.014673 8215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbf9a37cf51745948a2a7df6d52f094a" candidate_uuid: "655d707e276145a9a712ab8388e1d5d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d016d468d37b4edbab192d8c94d556bd"
I20260927 12:46:43.014731 8076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbf9a37cf51745948a2a7df6d52f094a" candidate_uuid: "655d707e276145a9a712ab8388e1d5d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b5ece2da9d94112a22a017cf744fdd4"
I20260927 12:46:43.014804 8215 raft_consensus.cc:3037] T fbf9a37cf51745948a2a7df6d52f094a P d016d468d37b4edbab192d8c94d556bd [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:46:43.014835 8076 raft_consensus.cc:3037] T fbf9a37cf51745948a2a7df6d52f094a P 7b5ece2da9d94112a22a017cf744fdd4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:46:43.016067 8215 raft_consensus.cc:2454] T fbf9a37cf51745948a2a7df6d52f094a P d016d468d37b4edbab192d8c94d556bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 655d707e276145a9a712ab8388e1d5d8 in term 1.
I20260927 12:46:43.016067 8076 raft_consensus.cc:2454] T fbf9a37cf51745948a2a7df6d52f094a P 7b5ece2da9d94112a22a017cf744fdd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 655d707e276145a9a712ab8388e1d5d8 in term 1.
I20260927 12:46:43.016395 8289 leader_election.cc:302] T fbf9a37cf51745948a2a7df6d52f094a P 655d707e276145a9a712ab8388e1d5d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 655d707e276145a9a712ab8388e1d5d8, 7b5ece2da9d94112a22a017cf744fdd4; no voters:
I20260927 12:46:43.016601 8437 raft_consensus.cc:2781] T fbf9a37cf51745948a2a7df6d52f094a P 655d707e276145a9a712ab8388e1d5d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:46:43.016812 8437 raft_consensus.cc:686] T fbf9a37cf51745948a2a7df6d52f094a P 655d707e276145a9a712ab8388e1d5d8 [term 1 LEADER]: Becoming Leader. State: Replica: 655d707e276145a9a712ab8388e1d5d8, State: Running, Role: LEADER
I20260927 12:46:43.017170 8437 consensus_queue.cc:237] T fbf9a37cf51745948a2a7df6d52f094a P 655d707e276145a9a712ab8388e1d5d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5ece2da9d94112a22a017cf744fdd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37315 } } peers { permanent_uuid: "d016d468d37b4edbab192d8c94d556bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34063 } } peers { permanent_uuid: "655d707e276145a9a712ab8388e1d5d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38265 } }
I20260927 12:46:43.020865 7918 catalog_manager.cc:5495] T fbf9a37cf51745948a2a7df6d52f094a P 655d707e276145a9a712ab8388e1d5d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 655d707e276145a9a712ab8388e1d5d8 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "655d707e276145a9a712ab8388e1d5d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5ece2da9d94112a22a017cf744fdd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d016d468d37b4edbab192d8c94d556bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "655d707e276145a9a712ab8388e1d5d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38265 } health_report { overall_health: HEALTHY } } }
I20260927 12:46:43.067999 8428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:43.064236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45982 (local address 127.0.98.3:38265)
0927 12:46:43.064357 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:43.064361 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:43.064375 (+ 14us) server_negotiation.cc:407] Connection header received
0927 12:46:43.064486 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:43.064488 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:43.064544 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:43.064625 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:46:43.064915 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:43.065538 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:43.066503 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:43.066803 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:43.066892 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:43.067131 (+ 239us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:46:43.067586 (+ 455us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:46:43.067751 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:43.067807 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 12:46:43.067860 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 12:46:43.073997 8076 raft_consensus.cc:1261] T fbf9a37cf51745948a2a7df6d52f094a P 7b5ece2da9d94112a22a017cf744fdd4 [term 1 FOLLOWER]: Refusing update from remote peer 655d707e276145a9a712ab8388e1d5d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:46:43.073997 8215 raft_consensus.cc:1261] T fbf9a37cf51745948a2a7df6d52f094a P d016d468d37b4edbab192d8c94d556bd [term 1 FOLLOWER]: Refusing update from remote peer 655d707e276145a9a712ab8388e1d5d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:46:43.074819 8440 consensus_queue.cc:1035] T fbf9a37cf51745948a2a7df6d52f094a P 655d707e276145a9a712ab8388e1d5d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d016d468d37b4edbab192d8c94d556bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:46:43.075027 8437 consensus_queue.cc:1035] T fbf9a37cf51745948a2a7df6d52f094a P 655d707e276145a9a712ab8388e1d5d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b5ece2da9d94112a22a017cf744fdd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:46:43.082597 8445 mvcc.cc:205] Tried to move back new op lower bound from 7333942079783645184 to 7333942079561412608. Current Snapshot: MvccSnapshot[applied={T|T < 7333942079783645184}]
I20260927 12:46:43.091907 7918 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49978:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:46:43.092058 7918 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49978:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:46:43.094684 7918 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 27371dbb6cd84fd7be84c8e671ae4f71: Sending DeleteTablet for 3 replicas of tablet fbf9a37cf51745948a2a7df6d52f094a
I20260927 12:46:43.095497 8056 tablet_service.cc:1514] Processing DeleteTablet for tablet fbf9a37cf51745948a2a7df6d52f094a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:46:43 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49210
I20260927 12:46:43.095494 8196 tablet_service.cc:1514] Processing DeleteTablet for tablet fbf9a37cf51745948a2a7df6d52f094a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:46:43 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53392
I20260927 12:46:43.095621 8336 tablet_service.cc:1514] Processing DeleteTablet for tablet fbf9a37cf51745948a2a7df6d52f094a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:46:43 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45966
I20260927 12:46:43.095985 8453 tablet_replica.cc:330] T fbf9a37cf51745948a2a7df6d52f094a P d016d468d37b4edbab192d8c94d556bd: stopping tablet replica
I20260927 12:46:43.096185 8454 tablet_replica.cc:330] T fbf9a37cf51745948a2a7df6d52f094a P 7b5ece2da9d94112a22a017cf744fdd4: stopping tablet replica
I20260927 12:46:43.096436 8453 raft_consensus.cc:2229] T fbf9a37cf51745948a2a7df6d52f094a P d016d468d37b4edbab192d8c94d556bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:46:43.096653 8454 raft_consensus.cc:2229] T fbf9a37cf51745948a2a7df6d52f094a P 7b5ece2da9d94112a22a017cf744fdd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:46:43.096769 7918 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49978:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 12:46:43.096982 8454 raft_consensus.cc:2258] T fbf9a37cf51745948a2a7df6d52f094a P 7b5ece2da9d94112a22a017cf744fdd4 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 12:46:43.097165 7918 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:46:43.097426 8455 tablet_replica.cc:330] T fbf9a37cf51745948a2a7df6d52f094a P 655d707e276145a9a712ab8388e1d5d8: stopping tablet replica
I20260927 12:46:43.097697 8455 raft_consensus.cc:2229] T fbf9a37cf51745948a2a7df6d52f094a P 655d707e276145a9a712ab8388e1d5d8 [term 1 LEADER]: Raft consensus shutting down.
I20260927 12:46:43.097980 8453 raft_consensus.cc:2258] T fbf9a37cf51745948a2a7df6d52f094a P d016d468d37b4edbab192d8c94d556bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:46:43.098129 8455 raft_consensus.cc:2258] T fbf9a37cf51745948a2a7df6d52f094a P 655d707e276145a9a712ab8388e1d5d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:46:43.099342 8454 ts_tablet_manager.cc:1905] T fbf9a37cf51745948a2a7df6d52f094a P 7b5ece2da9d94112a22a017cf744fdd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:46:43.099535 8455 ts_tablet_manager.cc:1905] T fbf9a37cf51745948a2a7df6d52f094a P 655d707e276145a9a712ab8388e1d5d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:46:43.101495 8454 ts_tablet_manager.cc:1918] T fbf9a37cf51745948a2a7df6d52f094a P 7b5ece2da9d94112a22a017cf744fdd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:46:43.101596 8454 log.cc:1196] T fbf9a37cf51745948a2a7df6d52f094a P 7b5ece2da9d94112a22a017cf744fdd4: Deleting WAL directory at /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/ts-0/wal/wals/fbf9a37cf51745948a2a7df6d52f094a
I20260927 12:46:43.101850 8454 ts_tablet_manager.cc:1939] T fbf9a37cf51745948a2a7df6d52f094a P 7b5ece2da9d94112a22a017cf744fdd4: Deleting consensus metadata
I20260927 12:46:43.102072 8455 ts_tablet_manager.cc:1918] T fbf9a37cf51745948a2a7df6d52f094a P 655d707e276145a9a712ab8388e1d5d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:46:43.102161 8455 log.cc:1196] T fbf9a37cf51745948a2a7df6d52f094a P 655d707e276145a9a712ab8388e1d5d8: Deleting WAL directory at /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/ts-2/wal/wals/fbf9a37cf51745948a2a7df6d52f094a
I20260927 12:46:43.102432 8455 ts_tablet_manager.cc:1939] T fbf9a37cf51745948a2a7df6d52f094a P 655d707e276145a9a712ab8388e1d5d8: Deleting consensus metadata
I20260927 12:46:43.103070 7905 catalog_manager.cc:4841] TS 7b5ece2da9d94112a22a017cf744fdd4 (127.0.98.1:37315): tablet fbf9a37cf51745948a2a7df6d52f094a (table test-table [id=d16653f03bca48f992c01911d7931c92]) successfully deleted
I20260927 12:46:43.103466 7903 catalog_manager.cc:4841] TS 655d707e276145a9a712ab8388e1d5d8 (127.0.98.3:38265): tablet fbf9a37cf51745948a2a7df6d52f094a (table test-table [id=d16653f03bca48f992c01911d7931c92]) successfully deleted
I20260927 12:46:43.104650 8453 ts_tablet_manager.cc:1905] T fbf9a37cf51745948a2a7df6d52f094a P d016d468d37b4edbab192d8c94d556bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:46:43.106204 8056 tablet_service.cc:1467] Processing CreateTablet for tablet 1c252a61ab144a4ba7ee8bcb2413dcf0 (DEFAULT_TABLE table=test-table [id=3e2b5947a1b2434f961692fcaefa65a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:46:43.106384 8196 tablet_service.cc:1467] Processing CreateTablet for tablet 1c252a61ab144a4ba7ee8bcb2413dcf0 (DEFAULT_TABLE table=test-table [id=3e2b5947a1b2434f961692fcaefa65a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:46:43.106460 8056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c252a61ab144a4ba7ee8bcb2413dcf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:46:43.106441 8336 tablet_service.cc:1467] Processing CreateTablet for tablet 1c252a61ab144a4ba7ee8bcb2413dcf0 (DEFAULT_TABLE table=test-table [id=3e2b5947a1b2434f961692fcaefa65a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:46:43.106654 8196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c252a61ab144a4ba7ee8bcb2413dcf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:46:43.106654 8336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c252a61ab144a4ba7ee8bcb2413dcf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:46:43.107182 8453 ts_tablet_manager.cc:1918] T fbf9a37cf51745948a2a7df6d52f094a P d016d468d37b4edbab192d8c94d556bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:46:43.107262 8453 log.cc:1196] T fbf9a37cf51745948a2a7df6d52f094a P d016d468d37b4edbab192d8c94d556bd: Deleting WAL directory at /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/ts-1/wal/wals/fbf9a37cf51745948a2a7df6d52f094a
I20260927 12:46:43.107538 8453 ts_tablet_manager.cc:1939] T fbf9a37cf51745948a2a7df6d52f094a P d016d468d37b4edbab192d8c94d556bd: Deleting consensus metadata
I20260927 12:46:43.108677 8432 tablet_bootstrap.cc:492] T 1c252a61ab144a4ba7ee8bcb2413dcf0 P d016d468d37b4edbab192d8c94d556bd: Bootstrap starting.
I20260927 12:46:43.108678 8430 tablet_bootstrap.cc:492] T 1c252a61ab144a4ba7ee8bcb2413dcf0 P 655d707e276145a9a712ab8388e1d5d8: Bootstrap starting.
I20260927 12:46:43.108738 8431 tablet_bootstrap.cc:492] T 1c252a61ab144a4ba7ee8bcb2413dcf0 P 7b5ece2da9d94112a22a017cf744fdd4: Bootstrap starting.
I20260927 12:46:43.109580 8432 tablet_bootstrap.cc:654] T 1c252a61ab144a4ba7ee8bcb2413dcf0 P d016d468d37b4edbab192d8c94d556bd: Neither blocks nor log segments found. Creating new log.
I20260927 12:46:43.109580 8430 tablet_bootstrap.cc:654] T 1c252a61ab144a4ba7ee8bcb2413dcf0 P 655d707e276145a9a712ab8388e1d5d8: Neither blocks nor log segments found. Creating new log.
I20260927 12:46:43.109630 8431 tablet_bootstrap.cc:654] T 1c252a61ab144a4ba7ee8bcb2413dcf0 P 7b5ece2da9d94112a22a017cf744fdd4: Neither blocks nor log segments found. Creating new log.
I20260927 12:46:43.110357 7904 catalog_manager.cc:4841] TS d016d468d37b4edbab192d8c94d556bd (127.0.98.2:34063): tablet fbf9a37cf51745948a2a7df6d52f094a (table test-table [id=d16653f03bca48f992c01911d7931c92]) successfully deleted
I20260927 12:46:43.111123 8432 tablet_bootstrap.cc:492] T 1c252a61ab144a4ba7ee8bcb2413dcf0 P d016d468d37b4edbab192d8c94d556bd: No bootstrap required, opened a new log
I20260927 12:46:43.111140 8431 tablet_bootstrap.cc:492] T 1c252a61ab144a4ba7ee8bcb2413dcf0 P 7b5ece2da9d94112a22a017cf744fdd4: No bootstrap required, opened a new log
I20260927 12:46:43.111167 8430 tablet_bootstrap.cc:492] T 1c252a61ab144a4ba7ee8bcb2413dcf0 P 655d707e276145a9a712ab8388e1d5d8: No bootstrap required, opened a new log
I20260927 12:46:43.111205 8432 ts_tablet_manager.cc:1397] T 1c252a61ab144a4ba7ee8bcb2413dcf0 P d016d468d37b4edbab192d8c94d556bd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 12:46:43.111205 8431 ts_tablet_manager.cc:1397] T 1c252a61ab144a4ba7ee8bcb2413dcf0 P 7b5ece2da9d94112a22a017cf744fdd4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 12:46:43.111236 8430 ts_tablet_manager.cc:1397] T 1c252a61ab144a4ba7ee8bcb2413dcf0 P 655d707e276145a9a712ab8388e1d5d8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 12:46:43.111677 8430 raft_consensus.cc:348] T 1c252a61ab144a4ba7ee8bcb2413dcf0 P 655d707e276145a9a712ab8388e1d5d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5ece2da9d94112a22a017cf744fdd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37315 } } peers { permanent_uuid: "d016d468d37b4edbab192d8c94d556bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34063 } } peers { permanent_uuid: "655d707e276145a9a712ab8388e1d5d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38265 } }
I20260927 12:46:43.111686 8432 raft_consensus.cc:348] T 1c252a61ab144a4ba7ee8bcb2413dcf0 P d016d468d37b4edbab192d8c94d556bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5ece2da9d94112a22a017cf744fdd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37315 } } peers { permanent_uuid: "d016d468d37b4edbab192d8c94d556bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34063 } } peers { permanent_uuid: "655d707e276145a9a712ab8388e1d5d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38265 } }
I20260927 12:46:43.111797 8430 raft_consensus.cc:374] T 1c252a61ab144a4ba7ee8bcb2413dcf0 P 655d707e276145a9a712ab8388e1d5d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:46:43.111816 8432 raft_consensus.cc:374] T 1c252a61ab144a4ba7ee8bcb2413dcf0 P d016d468d37b4edbab192d8c94d556bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:46:43.111830 8430 raft_consensus.cc:729] T 1c252a61ab144a4ba7ee8bcb2413dcf0 P 655d707e276145a9a712ab8388e1d5d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 655d707e276145a9a712ab8388e1d5d8, State: Initialized, Role: FOLLOWER
I20260927 12:46:43.111851 8432 raft_consensus.cc:729] T 1c252a61ab144a4ba7ee8bcb2413dcf0 P d016d468d37b4edbab192d8c94d556bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d016d468d37b4edbab192d8c94d556bd, State: Initialized, Role: FOLLOWER
I20260927 12:46:43.111929 8431 raft_consensus.cc:348] T 1c252a61ab144a4ba7ee8bcb2413dcf0 P 7b5ece2da9d94112a22a017cf744fdd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5ece2da9d94112a22a017cf744fdd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37315 } } peers { permanent_uuid: "d016d468d37b4edbab192d8c94d556bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34063 } } peers { permanent_uuid: "655d707e276145a9a712ab8388e1d5d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38265 } }
I20260927 12:46:43.111951 8430 consensus_queue.cc:260] T 1c252a61ab144a4ba7ee8bcb2413dcf0 P 655d707e276145a9a712ab8388e1d5d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5ece2da9d94112a22a017cf744fdd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37315 } } peers { permanent_uuid: "d016d468d37b4edbab192d8c94d556bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34063 } } peers { permanent_uuid: "655d707e276145a9a712ab8388e1d5d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38265 } }
I20260927 12:46:43.112023 8431 raft_consensus.cc:374] T 1c252a61ab144a4ba7ee8bcb2413dcf0 P 7b5ece2da9d94112a22a017cf744fdd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:46:43.112054 8431 raft_consensus.cc:729] T 1c252a61ab144a4ba7ee8bcb2413dcf0 P 7b5ece2da9d94112a22a017cf744fdd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b5ece2da9d94112a22a017cf744fdd4, State: Initialized, Role: FOLLOWER
I20260927 12:46:43.112195 8430 ts_tablet_manager.cc:1428] T 1c252a61ab144a4ba7ee8bcb2413dcf0 P 655d707e276145a9a712ab8388e1d5d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 12:46:43.112159 8431 consensus_queue.cc:260] T 1c252a61ab144a4ba7ee8bcb2413dcf0 P 7b5ece2da9d94112a22a017cf744fdd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5ece2da9d94112a22a017cf744fdd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37315 } } peers { permanent_uuid: "d016d468d37b4edbab192d8c94d556bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34063 } } peers { permanent_uuid: "655d707e276145a9a712ab8388e1d5d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38265 } }
I20260927 12:46:43.112413 8431 ts_tablet_manager.cc:1428] T 1c252a61ab144a4ba7ee8bcb2413dcf0 P 7b5ece2da9d94112a22a017cf744fdd4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 12:46:43.112789 8432 consensus_queue.cc:260] T 1c252a61ab144a4ba7ee8bcb2413dcf0 P d016d468d37b4edbab192d8c94d556bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5ece2da9d94112a22a017cf744fdd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37315 } } peers { permanent_uuid: "d016d468d37b4edbab192d8c94d556bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34063 } } peers { permanent_uuid: "655d707e276145a9a712ab8388e1d5d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38265 } }
I20260927 12:46:43.115381 8432 ts_tablet_manager.cc:1428] T 1c252a61ab144a4ba7ee8bcb2413dcf0 P d016d468d37b4edbab192d8c94d556bd: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20260927 12:46:43.126551 8436 raft_consensus.cc:482] T 1c252a61ab144a4ba7ee8bcb2413dcf0 P 7b5ece2da9d94112a22a017cf744fdd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:46:43.126701 8436 raft_consensus.cc:504] T 1c252a61ab144a4ba7ee8bcb2413dcf0 P 7b5ece2da9d94112a22a017cf744fdd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5ece2da9d94112a22a017cf744fdd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37315 } } peers { permanent_uuid: "d016d468d37b4edbab192d8c94d556bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34063 } } peers { permanent_uuid: "655d707e276145a9a712ab8388e1d5d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38265 } }
I20260927 12:46:43.127923 8436 leader_election.cc:288] T 1c252a61ab144a4ba7ee8bcb2413dcf0 P 7b5ece2da9d94112a22a017cf744fdd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d016d468d37b4edbab192d8c94d556bd (127.0.98.2:34063), 655d707e276145a9a712ab8388e1d5d8 (127.0.98.3:38265)
I20260927 12:46:43.132305 8460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:43.128241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38265 (local address 127.0.98.1:58043)
0927 12:46:43.128760 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:43.128772 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:43.128871 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:43.129197 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:43.129200 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:43.129215 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:46:43.129337 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:43.129342 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:43.130309 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:43.130313 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:43.131142 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:43.131151 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:43.131873 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:43.131877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:43.131982 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:43.132007 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 12:46:43.132081 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 12:46:43.132150 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":98,"threads_started":1}
I20260927 12:46:43.132318 8428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:43.128578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58043 (local address 127.0.98.3:38265)
0927 12:46:43.128714 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:43.128719 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:43.128848 (+ 129us) server_negotiation.cc:407] Connection header received
0927 12:46:43.128990 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:43.128993 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:43.129038 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:43.129081 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:46:43.129448 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:43.130201 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:43.131271 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:43.131739 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:43.131852 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:43.131934 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:43.132105 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 12:46:43.132183 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 12:46:43.132853 8459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:43.128144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34063 (local address 127.0.98.1:42503)
0927 12:46:43.128472 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:43.128488 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:43.128639 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:43.128948 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:43.128951 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:43.128969 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:46:43.129097 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:43.129103 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:43.130167 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:43.130171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:43.131488 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:43.131498 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:43.132406 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:43.132410 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:43.132515 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:43.132544 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 12:46:43.132627 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 12:46:43.132695 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":34,"thread_start_us":105,"threads_started":1}
I20260927 12:46:43.132843 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:43.128221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42503 (local address 127.0.98.2:34063)
0927 12:46:43.128482 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:43.128488 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:43.128552 (+ 64us) server_negotiation.cc:407] Connection header received
0927 12:46:43.128785 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:43.128789 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:43.128845 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:43.128934 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:46:43.129218 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:43.129993 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:43.131617 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:43.132157 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:43.132264 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:43.132328 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:43.132643 (+ 315us) server_negotiation.cc:299] Negotiation successful
0927 12:46:43.132720 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159}
I20260927 12:46:43.133121 8356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c252a61ab144a4ba7ee8bcb2413dcf0" candidate_uuid: "7b5ece2da9d94112a22a017cf744fdd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "655d707e276145a9a712ab8388e1d5d8" is_pre_election: true
I20260927 12:46:43.133287 8215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c252a61ab144a4ba7ee8bcb2413dcf0" candidate_uuid: "7b5ece2da9d94112a22a017cf744fdd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d016d468d37b4edbab192d8c94d556bd" is_pre_election: true
I20260927 12:46:43.133370 8356 raft_consensus.cc:2454] T 1c252a61ab144a4ba7ee8bcb2413dcf0 P 655d707e276145a9a712ab8388e1d5d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b5ece2da9d94112a22a017cf744fdd4 in term 0.
I20260927 12:46:43.133436 8215 raft_consensus.cc:2454] T 1c252a61ab144a4ba7ee8bcb2413dcf0 P d016d468d37b4edbab192d8c94d556bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b5ece2da9d94112a22a017cf744fdd4 in term 0.
I20260927 12:46:43.133765 8007 leader_election.cc:302] T 1c252a61ab144a4ba7ee8bcb2413dcf0 P 7b5ece2da9d94112a22a017cf744fdd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 655d707e276145a9a712ab8388e1d5d8, 7b5ece2da9d94112a22a017cf744fdd4; no voters:
I20260927 12:46:43.134061 8436 raft_consensus.cc:2781] T 1c252a61ab144a4ba7ee8bcb2413dcf0 P 7b5ece2da9d94112a22a017cf744fdd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:46:43.134117 8436 raft_consensus.cc:482] T 1c252a61ab144a4ba7ee8bcb2413dcf0 P 7b5ece2da9d94112a22a017cf744fdd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:46:43.134153 8436 raft_consensus.cc:3037] T 1c252a61ab144a4ba7ee8bcb2413dcf0 P 7b5ece2da9d94112a22a017cf744fdd4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:46:43.135325 8436 raft_consensus.cc:504] T 1c252a61ab144a4ba7ee8bcb2413dcf0 P 7b5ece2da9d94112a22a017cf744fdd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5ece2da9d94112a22a017cf744fdd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37315 } } peers { permanent_uuid: "d016d468d37b4edbab192d8c94d556bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34063 } } peers { permanent_uuid: "655d707e276145a9a712ab8388e1d5d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38265 } }
I20260927 12:46:43.135687 8436 leader_election.cc:288] T 1c252a61ab144a4ba7ee8bcb2413dcf0 P 7b5ece2da9d94112a22a017cf744fdd4 [CANDIDATE]: Term 1 election: Requested vote from peers d016d468d37b4edbab192d8c94d556bd (127.0.98.2:34063), 655d707e276145a9a712ab8388e1d5d8 (127.0.98.3:38265)
I20260927 12:46:43.136045 8356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c252a61ab144a4ba7ee8bcb2413dcf0" candidate_uuid: "7b5ece2da9d94112a22a017cf744fdd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "655d707e276145a9a712ab8388e1d5d8"
I20260927 12:46:43.136059 8215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c252a61ab144a4ba7ee8bcb2413dcf0" candidate_uuid: "7b5ece2da9d94112a22a017cf744fdd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d016d468d37b4edbab192d8c94d556bd"
I20260927 12:46:43.136158 8356 raft_consensus.cc:3037] T 1c252a61ab144a4ba7ee8bcb2413dcf0 P 655d707e276145a9a712ab8388e1d5d8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:46:43.136169 8215 raft_consensus.cc:3037] T 1c252a61ab144a4ba7ee8bcb2413dcf0 P d016d468d37b4edbab192d8c94d556bd [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:46:43.137225 8356 raft_consensus.cc:2454] T 1c252a61ab144a4ba7ee8bcb2413dcf0 P 655d707e276145a9a712ab8388e1d5d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b5ece2da9d94112a22a017cf744fdd4 in term 1.
I20260927 12:46:43.137251 8215 raft_consensus.cc:2454] T 1c252a61ab144a4ba7ee8bcb2413dcf0 P d016d468d37b4edbab192d8c94d556bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b5ece2da9d94112a22a017cf744fdd4 in term 1.
I20260927 12:46:43.137524 8008 leader_election.cc:302] T 1c252a61ab144a4ba7ee8bcb2413dcf0 P 7b5ece2da9d94112a22a017cf744fdd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b5ece2da9d94112a22a017cf744fdd4, d016d468d37b4edbab192d8c94d556bd; no voters:
I20260927 12:46:43.137701 8436 raft_consensus.cc:2781] T 1c252a61ab144a4ba7ee8bcb2413dcf0 P 7b5ece2da9d94112a22a017cf744fdd4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:46:43.137915 8436 raft_consensus.cc:686] T 1c252a61ab144a4ba7ee8bcb2413dcf0 P 7b5ece2da9d94112a22a017cf744fdd4 [term 1 LEADER]: Becoming Leader. State: Replica: 7b5ece2da9d94112a22a017cf744fdd4, State: Running, Role: LEADER
I20260927 12:46:43.138207 8436 consensus_queue.cc:237] T 1c252a61ab144a4ba7ee8bcb2413dcf0 P 7b5ece2da9d94112a22a017cf744fdd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5ece2da9d94112a22a017cf744fdd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37315 } } peers { permanent_uuid: "d016d468d37b4edbab192d8c94d556bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34063 } } peers { permanent_uuid: "655d707e276145a9a712ab8388e1d5d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38265 } }
I20260927 12:46:43.141012 7919 catalog_manager.cc:5495] T 1c252a61ab144a4ba7ee8bcb2413dcf0 P 7b5ece2da9d94112a22a017cf744fdd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b5ece2da9d94112a22a017cf744fdd4 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "7b5ece2da9d94112a22a017cf744fdd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5ece2da9d94112a22a017cf744fdd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d016d468d37b4edbab192d8c94d556bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "655d707e276145a9a712ab8388e1d5d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38265 } health_report { overall_health: UNKNOWN } } }
Sep 27 12:46:43 dist-test-slave-lkn4 krb5kdc[7874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513201, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 12:46:43.192974 7968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:43.183857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49992 (local address 127.0.98.62:42721)
0927 12:46:43.183995 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:43.184001 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:43.184212 (+ 211us) server_negotiation.cc:407] Connection header received
0927 12:46:43.184324 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:43.184326 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:43.184366 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:43.184439 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:43.185276 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:43.185810 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:43.186937 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:43.187195 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:43.187275 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:43.189760 (+ 2485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:43.189763 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:43.189765 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:43.189786 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:43.191204 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:43.191699 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:43.191702 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:43.191703 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:43.191776 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:43.192029 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:43.192031 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:43.192033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:43.192217 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:43.192288 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:43.192628 (+ 340us) server_negotiation.cc:299] Negotiation successful
0927 12:46:43.192802 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":71,"server-negotiator.queue_time_us":54}
I20260927 12:46:43.192991 8468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:43.183776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42721 (local address 127.0.0.1:49992)
0927 12:46:43.184092 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:43.184108 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:43.184235 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:43.184486 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:43.184489 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:43.185030 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:43.185146 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:43.185153 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:43.185917 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:43.185921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:43.186817 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:43.186828 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:43.187303 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:43.187306 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:43.187387 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:43.187954 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:43.187967 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:43.189555 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:43.191301 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:43.191310 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:43.191316 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:43.191598 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:43.191854 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:43.191857 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:43.191860 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:43.191951 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:43.192308 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:43.192315 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:43.192450 (+ 135us) client_negotiation.cc:769] Sending connection context
0927 12:46:43.192567 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 12:46:43.192739 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":94,"threads_started":1}
I20260927 12:46:43.196843 7919 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49992:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 12:46:43.197551 7919 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:46:43.202512 8056 tablet_service.cc:1467] Processing CreateTablet for tablet af868c8726714d6cac096b135b6c34cd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=38774b6c26ef4d8d8fb243e1df900c80]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 12:46:43.202700 8336 tablet_service.cc:1467] Processing CreateTablet for tablet af868c8726714d6cac096b135b6c34cd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=38774b6c26ef4d8d8fb243e1df900c80]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 12:46:43.202934 8056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af868c8726714d6cac096b135b6c34cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:46:43.203032 8336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af868c8726714d6cac096b135b6c34cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:46:43.205199 8430 tablet_bootstrap.cc:492] T af868c8726714d6cac096b135b6c34cd P 655d707e276145a9a712ab8388e1d5d8: Bootstrap starting.
I20260927 12:46:43.205514 8196 tablet_service.cc:1467] Processing CreateTablet for tablet af868c8726714d6cac096b135b6c34cd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=38774b6c26ef4d8d8fb243e1df900c80]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 12:46:43.205819 8431 tablet_bootstrap.cc:492] T af868c8726714d6cac096b135b6c34cd P 7b5ece2da9d94112a22a017cf744fdd4: Bootstrap starting.
I20260927 12:46:43.205894 8196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af868c8726714d6cac096b135b6c34cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:46:43.206143 8430 tablet_bootstrap.cc:654] T af868c8726714d6cac096b135b6c34cd P 655d707e276145a9a712ab8388e1d5d8: Neither blocks nor log segments found. Creating new log.
I20260927 12:46:43.206720 8431 tablet_bootstrap.cc:654] T af868c8726714d6cac096b135b6c34cd P 7b5ece2da9d94112a22a017cf744fdd4: Neither blocks nor log segments found. Creating new log.
I20260927 12:46:43.207518 8430 tablet_bootstrap.cc:492] T af868c8726714d6cac096b135b6c34cd P 655d707e276145a9a712ab8388e1d5d8: No bootstrap required, opened a new log
I20260927 12:46:43.207593 8430 ts_tablet_manager.cc:1397] T af868c8726714d6cac096b135b6c34cd P 655d707e276145a9a712ab8388e1d5d8: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260927 12:46:43.208024 8430 raft_consensus.cc:348] T af868c8726714d6cac096b135b6c34cd P 655d707e276145a9a712ab8388e1d5d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5ece2da9d94112a22a017cf744fdd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37315 } } peers { permanent_uuid: "d016d468d37b4edbab192d8c94d556bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34063 } } peers { permanent_uuid: "655d707e276145a9a712ab8388e1d5d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38265 } }
I20260927 12:46:43.208127 8430 raft_consensus.cc:374] T af868c8726714d6cac096b135b6c34cd P 655d707e276145a9a712ab8388e1d5d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:46:43.208166 8430 raft_consensus.cc:729] T af868c8726714d6cac096b135b6c34cd P 655d707e276145a9a712ab8388e1d5d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 655d707e276145a9a712ab8388e1d5d8, State: Initialized, Role: FOLLOWER
I20260927 12:46:43.208287 8430 consensus_queue.cc:260] T af868c8726714d6cac096b135b6c34cd P 655d707e276145a9a712ab8388e1d5d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5ece2da9d94112a22a017cf744fdd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37315 } } peers { permanent_uuid: "d016d468d37b4edbab192d8c94d556bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34063 } } peers { permanent_uuid: "655d707e276145a9a712ab8388e1d5d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38265 } }
I20260927 12:46:43.208529 8430 ts_tablet_manager.cc:1428] T af868c8726714d6cac096b135b6c34cd P 655d707e276145a9a712ab8388e1d5d8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 12:46:43.208508 8440 tablet_replica.cc:439] T af868c8726714d6cac096b135b6c34cd P 655d707e276145a9a712ab8388e1d5d8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5ece2da9d94112a22a017cf744fdd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37315 } } peers { permanent_uuid: "d016d468d37b4edbab192d8c94d556bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34063 } } peers { permanent_uuid: "655d707e276145a9a712ab8388e1d5d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38265 } } }
I20260927 12:46:43.208635 8440 tablet_replica.cc:442] T af868c8726714d6cac096b135b6c34cd P 655d707e276145a9a712ab8388e1d5d8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 12:46:43.209389 8432 tablet_bootstrap.cc:492] T af868c8726714d6cac096b135b6c34cd P d016d468d37b4edbab192d8c94d556bd: Bootstrap starting.
I20260927 12:46:43.209884 8431 tablet_bootstrap.cc:492] T af868c8726714d6cac096b135b6c34cd P 7b5ece2da9d94112a22a017cf744fdd4: No bootstrap required, opened a new log
I20260927 12:46:43.209966 8431 ts_tablet_manager.cc:1397] T af868c8726714d6cac096b135b6c34cd P 7b5ece2da9d94112a22a017cf744fdd4: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 12:46:43.210327 8432 tablet_bootstrap.cc:654] T af868c8726714d6cac096b135b6c34cd P d016d468d37b4edbab192d8c94d556bd: Neither blocks nor log segments found. Creating new log.
I20260927 12:46:43.210445 8431 raft_consensus.cc:348] T af868c8726714d6cac096b135b6c34cd P 7b5ece2da9d94112a22a017cf744fdd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5ece2da9d94112a22a017cf744fdd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37315 } } peers { permanent_uuid: "d016d468d37b4edbab192d8c94d556bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34063 } } peers { permanent_uuid: "655d707e276145a9a712ab8388e1d5d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38265 } }
I20260927 12:46:43.210568 8431 raft_consensus.cc:374] T af868c8726714d6cac096b135b6c34cd P 7b5ece2da9d94112a22a017cf744fdd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:46:43.210599 8431 raft_consensus.cc:729] T af868c8726714d6cac096b135b6c34cd P 7b5ece2da9d94112a22a017cf744fdd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b5ece2da9d94112a22a017cf744fdd4, State: Initialized, Role: FOLLOWER
I20260927 12:46:43.210721 8431 consensus_queue.cc:260] T af868c8726714d6cac096b135b6c34cd P 7b5ece2da9d94112a22a017cf744fdd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5ece2da9d94112a22a017cf744fdd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37315 } } peers { permanent_uuid: "d016d468d37b4edbab192d8c94d556bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34063 } } peers { permanent_uuid: "655d707e276145a9a712ab8388e1d5d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38265 } }
I20260927 12:46:43.210965 8436 tablet_replica.cc:439] T af868c8726714d6cac096b135b6c34cd P 7b5ece2da9d94112a22a017cf744fdd4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5ece2da9d94112a22a017cf744fdd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37315 } } peers { permanent_uuid: "d016d468d37b4edbab192d8c94d556bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34063 } } peers { permanent_uuid: "655d707e276145a9a712ab8388e1d5d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38265 } } }
I20260927 12:46:43.211103 8436 tablet_replica.cc:442] T af868c8726714d6cac096b135b6c34cd P 7b5ece2da9d94112a22a017cf744fdd4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 12:46:43.210976 8431 ts_tablet_manager.cc:1428] T af868c8726714d6cac096b135b6c34cd P 7b5ece2da9d94112a22a017cf744fdd4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 12:46:43.212041 8432 tablet_bootstrap.cc:492] T af868c8726714d6cac096b135b6c34cd P d016d468d37b4edbab192d8c94d556bd: No bootstrap required, opened a new log
I20260927 12:46:43.212119 8432 ts_tablet_manager.cc:1397] T af868c8726714d6cac096b135b6c34cd P d016d468d37b4edbab192d8c94d556bd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 12:46:43.212514 8432 raft_consensus.cc:348] T af868c8726714d6cac096b135b6c34cd P d016d468d37b4edbab192d8c94d556bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5ece2da9d94112a22a017cf744fdd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37315 } } peers { permanent_uuid: "d016d468d37b4edbab192d8c94d556bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34063 } } peers { permanent_uuid: "655d707e276145a9a712ab8388e1d5d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38265 } }
I20260927 12:46:43.212615 8432 raft_consensus.cc:374] T af868c8726714d6cac096b135b6c34cd P d016d468d37b4edbab192d8c94d556bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:46:43.212644 8432 raft_consensus.cc:729] T af868c8726714d6cac096b135b6c34cd P d016d468d37b4edbab192d8c94d556bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d016d468d37b4edbab192d8c94d556bd, State: Initialized, Role: FOLLOWER
I20260927 12:46:43.212766 8432 consensus_queue.cc:260] T af868c8726714d6cac096b135b6c34cd P d016d468d37b4edbab192d8c94d556bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5ece2da9d94112a22a017cf744fdd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37315 } } peers { permanent_uuid: "d016d468d37b4edbab192d8c94d556bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34063 } } peers { permanent_uuid: "655d707e276145a9a712ab8388e1d5d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38265 } }
I20260927 12:46:43.213006 8432 ts_tablet_manager.cc:1428] T af868c8726714d6cac096b135b6c34cd P d016d468d37b4edbab192d8c94d556bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 12:46:43.213203 8438 tablet_replica.cc:439] T af868c8726714d6cac096b135b6c34cd P d016d468d37b4edbab192d8c94d556bd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5ece2da9d94112a22a017cf744fdd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37315 } } peers { permanent_uuid: "d016d468d37b4edbab192d8c94d556bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34063 } } peers { permanent_uuid: "655d707e276145a9a712ab8388e1d5d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38265 } } }
I20260927 12:46:43.213304 8438 tablet_replica.cc:442] T af868c8726714d6cac096b135b6c34cd P d016d468d37b4edbab192d8c94d556bd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 12:46:43.295379 8436 raft_consensus.cc:482] T af868c8726714d6cac096b135b6c34cd P 7b5ece2da9d94112a22a017cf744fdd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:46:43.295537 8436 raft_consensus.cc:504] T af868c8726714d6cac096b135b6c34cd P 7b5ece2da9d94112a22a017cf744fdd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5ece2da9d94112a22a017cf744fdd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37315 } } peers { permanent_uuid: "d016d468d37b4edbab192d8c94d556bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34063 } } peers { permanent_uuid: "655d707e276145a9a712ab8388e1d5d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38265 } }
I20260927 12:46:43.296010 8436 leader_election.cc:288] T af868c8726714d6cac096b135b6c34cd P 7b5ece2da9d94112a22a017cf744fdd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d016d468d37b4edbab192d8c94d556bd (127.0.98.2:34063), 655d707e276145a9a712ab8388e1d5d8 (127.0.98.3:38265)
I20260927 12:46:43.296504 8215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af868c8726714d6cac096b135b6c34cd" candidate_uuid: "7b5ece2da9d94112a22a017cf744fdd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d016d468d37b4edbab192d8c94d556bd" is_pre_election: true
I20260927 12:46:43.296504 8356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af868c8726714d6cac096b135b6c34cd" candidate_uuid: "7b5ece2da9d94112a22a017cf744fdd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "655d707e276145a9a712ab8388e1d5d8" is_pre_election: true
I20260927 12:46:43.296665 8215 raft_consensus.cc:2454] T af868c8726714d6cac096b135b6c34cd P d016d468d37b4edbab192d8c94d556bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b5ece2da9d94112a22a017cf744fdd4 in term 0.
I20260927 12:46:43.296665 8356 raft_consensus.cc:2454] T af868c8726714d6cac096b135b6c34cd P 655d707e276145a9a712ab8388e1d5d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b5ece2da9d94112a22a017cf744fdd4 in term 0.
I20260927 12:46:43.296958 8007 leader_election.cc:302] T af868c8726714d6cac096b135b6c34cd P 7b5ece2da9d94112a22a017cf744fdd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 655d707e276145a9a712ab8388e1d5d8, 7b5ece2da9d94112a22a017cf744fdd4, d016d468d37b4edbab192d8c94d556bd; no voters:
I20260927 12:46:43.297137 8436 raft_consensus.cc:2781] T af868c8726714d6cac096b135b6c34cd P 7b5ece2da9d94112a22a017cf744fdd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:46:43.297207 8436 raft_consensus.cc:482] T af868c8726714d6cac096b135b6c34cd P 7b5ece2da9d94112a22a017cf744fdd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:46:43.297236 8436 raft_consensus.cc:3037] T af868c8726714d6cac096b135b6c34cd P 7b5ece2da9d94112a22a017cf744fdd4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:46:43.298292 8436 raft_consensus.cc:504] T af868c8726714d6cac096b135b6c34cd P 7b5ece2da9d94112a22a017cf744fdd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5ece2da9d94112a22a017cf744fdd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37315 } } peers { permanent_uuid: "d016d468d37b4edbab192d8c94d556bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34063 } } peers { permanent_uuid: "655d707e276145a9a712ab8388e1d5d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38265 } }
I20260927 12:46:43.298619 8436 leader_election.cc:288] T af868c8726714d6cac096b135b6c34cd P 7b5ece2da9d94112a22a017cf744fdd4 [CANDIDATE]: Term 1 election: Requested vote from peers d016d468d37b4edbab192d8c94d556bd (127.0.98.2:34063), 655d707e276145a9a712ab8388e1d5d8 (127.0.98.3:38265)
I20260927 12:46:43.298991 8356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af868c8726714d6cac096b135b6c34cd" candidate_uuid: "7b5ece2da9d94112a22a017cf744fdd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "655d707e276145a9a712ab8388e1d5d8"
I20260927 12:46:43.299021 8215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af868c8726714d6cac096b135b6c34cd" candidate_uuid: "7b5ece2da9d94112a22a017cf744fdd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d016d468d37b4edbab192d8c94d556bd"
I20260927 12:46:43.299098 8356 raft_consensus.cc:3037] T af868c8726714d6cac096b135b6c34cd P 655d707e276145a9a712ab8388e1d5d8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:46:43.299146 8215 raft_consensus.cc:3037] T af868c8726714d6cac096b135b6c34cd P d016d468d37b4edbab192d8c94d556bd [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:46:43.300024 8356 raft_consensus.cc:2454] T af868c8726714d6cac096b135b6c34cd P 655d707e276145a9a712ab8388e1d5d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b5ece2da9d94112a22a017cf744fdd4 in term 1.
I20260927 12:46:43.300030 8215 raft_consensus.cc:2454] T af868c8726714d6cac096b135b6c34cd P d016d468d37b4edbab192d8c94d556bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b5ece2da9d94112a22a017cf744fdd4 in term 1.
I20260927 12:46:43.300285 8007 leader_election.cc:302] T af868c8726714d6cac096b135b6c34cd P 7b5ece2da9d94112a22a017cf744fdd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 655d707e276145a9a712ab8388e1d5d8, 7b5ece2da9d94112a22a017cf744fdd4; no voters:
I20260927 12:46:43.300455 8436 raft_consensus.cc:2781] T af868c8726714d6cac096b135b6c34cd P 7b5ece2da9d94112a22a017cf744fdd4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:46:43.300541 8436 raft_consensus.cc:686] T af868c8726714d6cac096b135b6c34cd P 7b5ece2da9d94112a22a017cf744fdd4 [term 1 LEADER]: Becoming Leader. State: Replica: 7b5ece2da9d94112a22a017cf744fdd4, State: Running, Role: LEADER
I20260927 12:46:43.300678 8436 consensus_queue.cc:237] T af868c8726714d6cac096b135b6c34cd P 7b5ece2da9d94112a22a017cf744fdd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5ece2da9d94112a22a017cf744fdd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37315 } } peers { permanent_uuid: "d016d468d37b4edbab192d8c94d556bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34063 } } peers { permanent_uuid: "655d707e276145a9a712ab8388e1d5d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38265 } }
I20260927 12:46:43.301335 8461 tablet_replica.cc:439] T af868c8726714d6cac096b135b6c34cd P 7b5ece2da9d94112a22a017cf744fdd4: TxnStatusTablet state changed. Reason: New leader 7b5ece2da9d94112a22a017cf744fdd4. Latest consensus state: current_term: 1 leader_uuid: "7b5ece2da9d94112a22a017cf744fdd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5ece2da9d94112a22a017cf744fdd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37315 } } peers { permanent_uuid: "d016d468d37b4edbab192d8c94d556bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34063 } } peers { permanent_uuid: "655d707e276145a9a712ab8388e1d5d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38265 } } }
I20260927 12:46:43.301447 8461 tablet_replica.cc:442] T af868c8726714d6cac096b135b6c34cd P 7b5ece2da9d94112a22a017cf744fdd4: This TxnStatusTablet replica's current role is: LEADER
I20260927 12:46:43.301703 8473 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 12:46:43.301865 8473 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 12:46:43.302376 7918 catalog_manager.cc:5495] T af868c8726714d6cac096b135b6c34cd P 7b5ece2da9d94112a22a017cf744fdd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b5ece2da9d94112a22a017cf744fdd4 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "7b5ece2da9d94112a22a017cf744fdd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5ece2da9d94112a22a017cf744fdd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d016d468d37b4edbab192d8c94d556bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "655d707e276145a9a712ab8388e1d5d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38265 } health_report { overall_health: UNKNOWN } } }
I20260927 12:46:43.315857 8468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:43.311582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37315 (local address 127.0.0.1:49224)
0927 12:46:43.311725 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:43.311741 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:43.311868 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:43.312256 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:43.312260 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:43.312271 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 12:46:43.312408 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:43.312415 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:43.313223 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:43.313227 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:43.314253 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:43.314266 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:43.314705 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:43.314709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:43.314798 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:43.314854 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 12:46:43.315518 (+ 664us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 12:46:43.315534 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 12:46:43.315604 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 12:46:43.315701 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 12:46:43.316087 8425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:43.311692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49224 (local address 127.0.98.1:37315)
0927 12:46:43.311800 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:43.311804 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:43.311817 (+ 13us) server_negotiation.cc:407] Connection header received
0927 12:46:43.312065 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:43.312068 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:43.312137 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:43.312221 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:46:43.312510 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:43.313118 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:43.314380 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:43.314600 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:43.314680 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:43.314980 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:46:43.315431 (+ 451us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:46:43.315524 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:43.315894 (+ 370us) server_negotiation.cc:299] Negotiation successful
0927 12:46:43.315956 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 12:46:43.318956 8215 raft_consensus.cc:1261] T af868c8726714d6cac096b135b6c34cd P d016d468d37b4edbab192d8c94d556bd [term 1 FOLLOWER]: Refusing update from remote peer 7b5ece2da9d94112a22a017cf744fdd4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:46:43.319162 8356 raft_consensus.cc:1261] T af868c8726714d6cac096b135b6c34cd P 655d707e276145a9a712ab8388e1d5d8 [term 1 FOLLOWER]: Refusing update from remote peer 7b5ece2da9d94112a22a017cf744fdd4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:46:43.319764 8436 consensus_queue.cc:1035] T af868c8726714d6cac096b135b6c34cd P 7b5ece2da9d94112a22a017cf744fdd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "655d707e276145a9a712ab8388e1d5d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:46:43.320147 8461 consensus_queue.cc:1035] T af868c8726714d6cac096b135b6c34cd P 7b5ece2da9d94112a22a017cf744fdd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d016d468d37b4edbab192d8c94d556bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:46:43.321394 8440 tablet_replica.cc:439] T af868c8726714d6cac096b135b6c34cd P 655d707e276145a9a712ab8388e1d5d8: TxnStatusTablet state changed. Reason: New leader 7b5ece2da9d94112a22a017cf744fdd4. Latest consensus state: current_term: 1 leader_uuid: "7b5ece2da9d94112a22a017cf744fdd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5ece2da9d94112a22a017cf744fdd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37315 } } peers { permanent_uuid: "d016d468d37b4edbab192d8c94d556bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34063 } } peers { permanent_uuid: "655d707e276145a9a712ab8388e1d5d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38265 } } }
I20260927 12:46:43.321506 8440 tablet_replica.cc:442] T af868c8726714d6cac096b135b6c34cd P 655d707e276145a9a712ab8388e1d5d8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 12:46:43.321710 8438 tablet_replica.cc:439] T af868c8726714d6cac096b135b6c34cd P d016d468d37b4edbab192d8c94d556bd: TxnStatusTablet state changed. Reason: New leader 7b5ece2da9d94112a22a017cf744fdd4. Latest consensus state: current_term: 1 leader_uuid: "7b5ece2da9d94112a22a017cf744fdd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5ece2da9d94112a22a017cf744fdd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37315 } } peers { permanent_uuid: "d016d468d37b4edbab192d8c94d556bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34063 } } peers { permanent_uuid: "655d707e276145a9a712ab8388e1d5d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38265 } } }
I20260927 12:46:43.321812 8438 tablet_replica.cc:442] T af868c8726714d6cac096b135b6c34cd P d016d468d37b4edbab192d8c94d556bd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 12:46:43.323601 8461 tablet_replica.cc:439] T af868c8726714d6cac096b135b6c34cd P 7b5ece2da9d94112a22a017cf744fdd4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7b5ece2da9d94112a22a017cf744fdd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5ece2da9d94112a22a017cf744fdd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37315 } } peers { permanent_uuid: "d016d468d37b4edbab192d8c94d556bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34063 } } peers { permanent_uuid: "655d707e276145a9a712ab8388e1d5d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38265 } } }
I20260927 12:46:43.323712 8461 tablet_replica.cc:442] T af868c8726714d6cac096b135b6c34cd P 7b5ece2da9d94112a22a017cf744fdd4: This TxnStatusTablet replica's current role is: LEADER
I20260927 12:46:43.325049 8444 mvcc.cc:205] Tried to move back new op lower bound from 7333942080789262336 to 7333942080721035264. Current Snapshot: MvccSnapshot[applied={T|T < 7333942080789262336 or (T in {7333942080789262336})}]
I20260927 12:46:43.325260 8438 tablet_replica.cc:439] T af868c8726714d6cac096b135b6c34cd P d016d468d37b4edbab192d8c94d556bd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7b5ece2da9d94112a22a017cf744fdd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5ece2da9d94112a22a017cf744fdd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37315 } } peers { permanent_uuid: "d016d468d37b4edbab192d8c94d556bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34063 } } peers { permanent_uuid: "655d707e276145a9a712ab8388e1d5d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38265 } } }
I20260927 12:46:43.325340 8438 tablet_replica.cc:442] T af868c8726714d6cac096b135b6c34cd P d016d468d37b4edbab192d8c94d556bd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 12:46:43.327109 8436 tablet_replica.cc:439] T af868c8726714d6cac096b135b6c34cd P 7b5ece2da9d94112a22a017cf744fdd4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7b5ece2da9d94112a22a017cf744fdd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5ece2da9d94112a22a017cf744fdd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37315 } } peers { permanent_uuid: "d016d468d37b4edbab192d8c94d556bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34063 } } peers { permanent_uuid: "655d707e276145a9a712ab8388e1d5d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38265 } } }
I20260927 12:46:43.327225 8436 tablet_replica.cc:442] T af868c8726714d6cac096b135b6c34cd P 7b5ece2da9d94112a22a017cf744fdd4: This TxnStatusTablet replica's current role is: LEADER
I20260927 12:46:43.328547 8440 tablet_replica.cc:439] T af868c8726714d6cac096b135b6c34cd P 655d707e276145a9a712ab8388e1d5d8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7b5ece2da9d94112a22a017cf744fdd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5ece2da9d94112a22a017cf744fdd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37315 } } peers { permanent_uuid: "d016d468d37b4edbab192d8c94d556bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34063 } } peers { permanent_uuid: "655d707e276145a9a712ab8388e1d5d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38265 } } }
I20260927 12:46:43.328663 8440 tablet_replica.cc:442] T af868c8726714d6cac096b135b6c34cd P 655d707e276145a9a712ab8388e1d5d8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 12:46:43.332686 8425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:43.329680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49236 (local address 127.0.98.1:37315)
0927 12:46:43.329784 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:43.329788 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:43.329849 (+ 61us) server_negotiation.cc:407] Connection header received
0927 12:46:43.330007 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:43.330010 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:43.330083 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:43.330142 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:46:43.330407 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:43.330925 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:43.331717 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:43.331945 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:43.332027 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:43.332272 (+ 245us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:46:43.332343 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:46:43.332410 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:43.332508 (+ 98us) server_negotiation.cc:299] Negotiation successful
0927 12:46:43.332562 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 12:46:43.343637 8425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:43.339869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59625 (local address 127.0.98.1:37315)
0927 12:46:43.339984 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:43.339988 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:43.340210 (+ 222us) server_negotiation.cc:407] Connection header received
0927 12:46:43.340359 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:43.340362 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:43.340406 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:43.340465 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:46:43.340751 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:43.341251 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:43.342542 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:43.342767 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:43.342874 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:43.343175 (+ 301us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:46:43.343244 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:46:43.343322 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:43.343423 (+ 101us) server_negotiation.cc:299] Negotiation successful
0927 12:46:43.343480 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":46}
I20260927 12:46:43.343657 8481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:43.339810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37315 (local address 127.0.98.1:59625)
0927 12:46:43.340116 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:43.340130 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:43.340250 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:43.340485 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:43.340488 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:43.340496 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 12:46:43.340643 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:43.340649 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:43.341345 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:43.341348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:43.342412 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:43.342423 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:43.342891 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:43.342894 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:43.342983 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:43.343026 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 12:46:43.343340 (+ 314us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 12:46:43.343345 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 12:46:43.343409 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 12:46:43.343511 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":20,"thread_start_us":86,"threads_started":1}
I20260927 12:46:43.350373 8215 raft_consensus.cc:1261] T 1c252a61ab144a4ba7ee8bcb2413dcf0 P d016d468d37b4edbab192d8c94d556bd [term 1 FOLLOWER]: Refusing update from remote peer 7b5ece2da9d94112a22a017cf744fdd4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:46:43.350443 8356 raft_consensus.cc:1261] T 1c252a61ab144a4ba7ee8bcb2413dcf0 P 655d707e276145a9a712ab8388e1d5d8 [term 1 FOLLOWER]: Refusing update from remote peer 7b5ece2da9d94112a22a017cf744fdd4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:46:43.350766 8478 consensus_queue.cc:1035] T 1c252a61ab144a4ba7ee8bcb2413dcf0 P 7b5ece2da9d94112a22a017cf744fdd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d016d468d37b4edbab192d8c94d556bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:46:43.350944 8436 consensus_queue.cc:1035] T 1c252a61ab144a4ba7ee8bcb2413dcf0 P 7b5ece2da9d94112a22a017cf744fdd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "655d707e276145a9a712ab8388e1d5d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:46:43.357275 8443 mvcc.cc:205] Tried to move back new op lower bound from 7333942080919322624 to 7333942080056725504. Current Snapshot: MvccSnapshot[applied={T|T < 7333942080919322624}]
I20260927 12:46:43.393332 7918 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49978:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:46:43.393476 7918 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49978:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:46:43.395514 7918 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 27371dbb6cd84fd7be84c8e671ae4f71: Sending DeleteTablet for 3 replicas of tablet 1c252a61ab144a4ba7ee8bcb2413dcf0
I20260927 12:46:43.396059 8055 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c252a61ab144a4ba7ee8bcb2413dcf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:46:43 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49210
I20260927 12:46:43.396134 8196 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c252a61ab144a4ba7ee8bcb2413dcf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:46:43 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53392
I20260927 12:46:43.396256 8454 tablet_replica.cc:330] T 1c252a61ab144a4ba7ee8bcb2413dcf0 P 7b5ece2da9d94112a22a017cf744fdd4: stopping tablet replica
I20260927 12:46:43.396358 8453 tablet_replica.cc:330] T 1c252a61ab144a4ba7ee8bcb2413dcf0 P d016d468d37b4edbab192d8c94d556bd: stopping tablet replica
I20260927 12:46:43.396371 8454 raft_consensus.cc:2229] T 1c252a61ab144a4ba7ee8bcb2413dcf0 P 7b5ece2da9d94112a22a017cf744fdd4 [term 1 LEADER]: Raft consensus shutting down.
I20260927 12:46:43.396363 8336 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c252a61ab144a4ba7ee8bcb2413dcf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:46:43 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45966
I20260927 12:46:43.396471 8453 raft_consensus.cc:2229] T 1c252a61ab144a4ba7ee8bcb2413dcf0 P d016d468d37b4edbab192d8c94d556bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:46:43.396525 8455 tablet_replica.cc:330] T 1c252a61ab144a4ba7ee8bcb2413dcf0 P 655d707e276145a9a712ab8388e1d5d8: stopping tablet replica
I20260927 12:46:43.396585 8453 raft_consensus.cc:2258] T 1c252a61ab144a4ba7ee8bcb2413dcf0 P d016d468d37b4edbab192d8c94d556bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:46:43.396620 8455 raft_consensus.cc:2229] T 1c252a61ab144a4ba7ee8bcb2413dcf0 P 655d707e276145a9a712ab8388e1d5d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:46:43.396695 8454 raft_consensus.cc:2258] T 1c252a61ab144a4ba7ee8bcb2413dcf0 P 7b5ece2da9d94112a22a017cf744fdd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:46:43.396718 8455 raft_consensus.cc:2258] T 1c252a61ab144a4ba7ee8bcb2413dcf0 P 655d707e276145a9a712ab8388e1d5d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:46:43.397573 8454 ts_tablet_manager.cc:1905] T 1c252a61ab144a4ba7ee8bcb2413dcf0 P 7b5ece2da9d94112a22a017cf744fdd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:46:43.399967 8453 ts_tablet_manager.cc:1905] T 1c252a61ab144a4ba7ee8bcb2413dcf0 P d016d468d37b4edbab192d8c94d556bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:46:43.400591 8454 ts_tablet_manager.cc:1918] T 1c252a61ab144a4ba7ee8bcb2413dcf0 P 7b5ece2da9d94112a22a017cf744fdd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 12:46:43.400662 8454 log.cc:1196] T 1c252a61ab144a4ba7ee8bcb2413dcf0 P 7b5ece2da9d94112a22a017cf744fdd4: Deleting WAL directory at /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/ts-0/wal/wals/1c252a61ab144a4ba7ee8bcb2413dcf0
I20260927 12:46:43.400877 8454 ts_tablet_manager.cc:1939] T 1c252a61ab144a4ba7ee8bcb2413dcf0 P 7b5ece2da9d94112a22a017cf744fdd4: Deleting consensus metadata
I20260927 12:46:43.401647 7905 catalog_manager.cc:4841] TS 7b5ece2da9d94112a22a017cf744fdd4 (127.0.98.1:37315): tablet 1c252a61ab144a4ba7ee8bcb2413dcf0 (table test-table [id=3e2b5947a1b2434f961692fcaefa65a6]) successfully deleted
I20260927 12:46:43.402787 8453 ts_tablet_manager.cc:1918] T 1c252a61ab144a4ba7ee8bcb2413dcf0 P d016d468d37b4edbab192d8c94d556bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 12:46:43.402869 8453 log.cc:1196] T 1c252a61ab144a4ba7ee8bcb2413dcf0 P d016d468d37b4edbab192d8c94d556bd: Deleting WAL directory at /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/ts-1/wal/wals/1c252a61ab144a4ba7ee8bcb2413dcf0
I20260927 12:46:43.403095 8453 ts_tablet_manager.cc:1939] T 1c252a61ab144a4ba7ee8bcb2413dcf0 P d016d468d37b4edbab192d8c94d556bd: Deleting consensus metadata
I20260927 12:46:43.403792 7904 catalog_manager.cc:4841] TS d016d468d37b4edbab192d8c94d556bd (127.0.98.2:34063): tablet 1c252a61ab144a4ba7ee8bcb2413dcf0 (table test-table [id=3e2b5947a1b2434f961692fcaefa65a6]) successfully deleted
I20260927 12:46:43.405741 8455 ts_tablet_manager.cc:1905] T 1c252a61ab144a4ba7ee8bcb2413dcf0 P 655d707e276145a9a712ab8388e1d5d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:46:43.408783 8455 ts_tablet_manager.cc:1918] T 1c252a61ab144a4ba7ee8bcb2413dcf0 P 655d707e276145a9a712ab8388e1d5d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 12:46:43.408879 8455 log.cc:1196] T 1c252a61ab144a4ba7ee8bcb2413dcf0 P 655d707e276145a9a712ab8388e1d5d8: Deleting WAL directory at /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/ts-2/wal/wals/1c252a61ab144a4ba7ee8bcb2413dcf0
I20260927 12:46:43.409103 8455 ts_tablet_manager.cc:1939] T 1c252a61ab144a4ba7ee8bcb2413dcf0 P 655d707e276145a9a712ab8388e1d5d8: Deleting consensus metadata
I20260927 12:46:43.409956 7903 catalog_manager.cc:4841] TS 655d707e276145a9a712ab8388e1d5d8 (127.0.98.3:38265): tablet 1c252a61ab144a4ba7ee8bcb2413dcf0 (table test-table [id=3e2b5947a1b2434f961692fcaefa65a6]) successfully deleted
Sep 27 12:46:43 dist-test-slave-lkn4 krb5kdc[7874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 12:46:43 dist-test-slave-lkn4 krb5kdc[7874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:46:43 dist-test-slave-lkn4 krb5kdc[7874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 12:46:43.425670 7968 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:46:43.419716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50000 (local address 127.0.98.62:42721)
0927 12:46:43.419831 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:43.419836 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:43.420097 (+ 261us) server_negotiation.cc:407] Connection header received
0927 12:46:43.420242 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:43.420245 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:43.420290 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:43.420363 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:43.421071 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:43.421592 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:43.422493 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:43.422710 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:43.422798 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:43.425544 (+ 2746us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50000: BlockingRecv error: recv got EOF from 127.0.0.1:50000 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 27 12:46:43 dist-test-slave-lkn4 krb5kdc[7874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 12:46:43.435858 7968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:43.427105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50006 (local address 127.0.98.62:42721)
0927 12:46:43.427206 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:43.427209 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:43.427453 (+ 244us) server_negotiation.cc:407] Connection header received
0927 12:46:43.427555 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:43.427557 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:43.427603 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:43.427673 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:43.428384 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:43.428922 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:43.430029 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:43.430301 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:43.430390 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:43.432913 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:43.432916 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:43.432918 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:43.432944 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:43.434316 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:43.434727 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:43.434731 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:43.434733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:43.434814 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:43.435060 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:43.435063 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:43.435065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:43.435252 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:43.435365 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:43.435586 (+ 221us) server_negotiation.cc:299] Negotiation successful
0927 12:46:43.435695 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 12:46:43.438365 7918 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50006:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:46:43.438832 7918 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:46:43.443398 8055 tablet_service.cc:1467] Processing CreateTablet for tablet e4940d47723c403cb069dc3ab28c84d7 (DEFAULT_TABLE table=test-table [id=13c3d8f2234643839cdf19c9baf639df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:46:43.443506 8196 tablet_service.cc:1467] Processing CreateTablet for tablet e4940d47723c403cb069dc3ab28c84d7 (DEFAULT_TABLE table=test-table [id=13c3d8f2234643839cdf19c9baf639df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:46:43.443704 8055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4940d47723c403cb069dc3ab28c84d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:46:43.443816 8196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4940d47723c403cb069dc3ab28c84d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:46:43.445691 8431 tablet_bootstrap.cc:492] T e4940d47723c403cb069dc3ab28c84d7 P 7b5ece2da9d94112a22a017cf744fdd4: Bootstrap starting.
I20260927 12:46:43.446333 8432 tablet_bootstrap.cc:492] T e4940d47723c403cb069dc3ab28c84d7 P d016d468d37b4edbab192d8c94d556bd: Bootstrap starting.
I20260927 12:46:43.446348 8336 tablet_service.cc:1467] Processing CreateTablet for tablet e4940d47723c403cb069dc3ab28c84d7 (DEFAULT_TABLE table=test-table [id=13c3d8f2234643839cdf19c9baf639df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:46:43.446604 8336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4940d47723c403cb069dc3ab28c84d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:46:43.446614 8431 tablet_bootstrap.cc:654] T e4940d47723c403cb069dc3ab28c84d7 P 7b5ece2da9d94112a22a017cf744fdd4: Neither blocks nor log segments found. Creating new log.
I20260927 12:46:43.447256 8432 tablet_bootstrap.cc:654] T e4940d47723c403cb069dc3ab28c84d7 P d016d468d37b4edbab192d8c94d556bd: Neither blocks nor log segments found. Creating new log.
I20260927 12:46:43.448401 8432 tablet_bootstrap.cc:492] T e4940d47723c403cb069dc3ab28c84d7 P d016d468d37b4edbab192d8c94d556bd: No bootstrap required, opened a new log
I20260927 12:46:43.448478 8432 ts_tablet_manager.cc:1397] T e4940d47723c403cb069dc3ab28c84d7 P d016d468d37b4edbab192d8c94d556bd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 12:46:43.448707 8431 tablet_bootstrap.cc:492] T e4940d47723c403cb069dc3ab28c84d7 P 7b5ece2da9d94112a22a017cf744fdd4: No bootstrap required, opened a new log
I20260927 12:46:43.448796 8431 ts_tablet_manager.cc:1397] T e4940d47723c403cb069dc3ab28c84d7 P 7b5ece2da9d94112a22a017cf744fdd4: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260927 12:46:43.448913 8430 tablet_bootstrap.cc:492] T e4940d47723c403cb069dc3ab28c84d7 P 655d707e276145a9a712ab8388e1d5d8: Bootstrap starting.
I20260927 12:46:43.448909 8432 raft_consensus.cc:348] T e4940d47723c403cb069dc3ab28c84d7 P d016d468d37b4edbab192d8c94d556bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5ece2da9d94112a22a017cf744fdd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37315 } } peers { permanent_uuid: "d016d468d37b4edbab192d8c94d556bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34063 } } peers { permanent_uuid: "655d707e276145a9a712ab8388e1d5d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38265 } }
I20260927 12:46:43.449016 8432 raft_consensus.cc:374] T e4940d47723c403cb069dc3ab28c84d7 P d016d468d37b4edbab192d8c94d556bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:46:43.449046 8432 raft_consensus.cc:729] T e4940d47723c403cb069dc3ab28c84d7 P d016d468d37b4edbab192d8c94d556bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d016d468d37b4edbab192d8c94d556bd, State: Initialized, Role: FOLLOWER
I20260927 12:46:43.449162 8432 consensus_queue.cc:260] T e4940d47723c403cb069dc3ab28c84d7 P d016d468d37b4edbab192d8c94d556bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5ece2da9d94112a22a017cf744fdd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37315 } } peers { permanent_uuid: "d016d468d37b4edbab192d8c94d556bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34063 } } peers { permanent_uuid: "655d707e276145a9a712ab8388e1d5d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38265 } }
I20260927 12:46:43.449249 8431 raft_consensus.cc:348] T e4940d47723c403cb069dc3ab28c84d7 P 7b5ece2da9d94112a22a017cf744fdd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5ece2da9d94112a22a017cf744fdd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37315 } } peers { permanent_uuid: "d016d468d37b4edbab192d8c94d556bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34063 } } peers { permanent_uuid: "655d707e276145a9a712ab8388e1d5d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38265 } }
I20260927 12:46:43.449363 8431 raft_consensus.cc:374] T e4940d47723c403cb069dc3ab28c84d7 P 7b5ece2da9d94112a22a017cf744fdd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:46:43.449398 8431 raft_consensus.cc:729] T e4940d47723c403cb069dc3ab28c84d7 P 7b5ece2da9d94112a22a017cf744fdd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b5ece2da9d94112a22a017cf744fdd4, State: Initialized, Role: FOLLOWER
I20260927 12:46:43.449424 8432 ts_tablet_manager.cc:1428] T e4940d47723c403cb069dc3ab28c84d7 P d016d468d37b4edbab192d8c94d556bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 12:46:43.449519 8431 consensus_queue.cc:260] T e4940d47723c403cb069dc3ab28c84d7 P 7b5ece2da9d94112a22a017cf744fdd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5ece2da9d94112a22a017cf744fdd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37315 } } peers { permanent_uuid: "d016d468d37b4edbab192d8c94d556bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34063 } } peers { permanent_uuid: "655d707e276145a9a712ab8388e1d5d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38265 } }
I20260927 12:46:43.449687 8430 tablet_bootstrap.cc:654] T e4940d47723c403cb069dc3ab28c84d7 P 655d707e276145a9a712ab8388e1d5d8: Neither blocks nor log segments found. Creating new log.
I20260927 12:46:43.449780 8431 ts_tablet_manager.cc:1428] T e4940d47723c403cb069dc3ab28c84d7 P 7b5ece2da9d94112a22a017cf744fdd4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 12:46:43.451573 8430 tablet_bootstrap.cc:492] T e4940d47723c403cb069dc3ab28c84d7 P 655d707e276145a9a712ab8388e1d5d8: No bootstrap required, opened a new log
I20260927 12:46:43.451674 8430 ts_tablet_manager.cc:1397] T e4940d47723c403cb069dc3ab28c84d7 P 655d707e276145a9a712ab8388e1d5d8: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260927 12:46:43.452013 8430 raft_consensus.cc:348] T e4940d47723c403cb069dc3ab28c84d7 P 655d707e276145a9a712ab8388e1d5d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5ece2da9d94112a22a017cf744fdd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37315 } } peers { permanent_uuid: "d016d468d37b4edbab192d8c94d556bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34063 } } peers { permanent_uuid: "655d707e276145a9a712ab8388e1d5d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38265 } }
I20260927 12:46:43.452122 8430 raft_consensus.cc:374] T e4940d47723c403cb069dc3ab28c84d7 P 655d707e276145a9a712ab8388e1d5d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:46:43.452155 8430 raft_consensus.cc:729] T e4940d47723c403cb069dc3ab28c84d7 P 655d707e276145a9a712ab8388e1d5d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 655d707e276145a9a712ab8388e1d5d8, State: Initialized, Role: FOLLOWER
I20260927 12:46:43.452273 8430 consensus_queue.cc:260] T e4940d47723c403cb069dc3ab28c84d7 P 655d707e276145a9a712ab8388e1d5d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5ece2da9d94112a22a017cf744fdd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37315 } } peers { permanent_uuid: "d016d468d37b4edbab192d8c94d556bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34063 } } peers { permanent_uuid: "655d707e276145a9a712ab8388e1d5d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38265 } }
I20260927 12:46:43.452543 8430 ts_tablet_manager.cc:1428] T e4940d47723c403cb069dc3ab28c84d7 P 655d707e276145a9a712ab8388e1d5d8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 12:46:43.699451 8438 raft_consensus.cc:482] T e4940d47723c403cb069dc3ab28c84d7 P d016d468d37b4edbab192d8c94d556bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:46:43.699597 8438 raft_consensus.cc:504] T e4940d47723c403cb069dc3ab28c84d7 P d016d468d37b4edbab192d8c94d556bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5ece2da9d94112a22a017cf744fdd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37315 } } peers { permanent_uuid: "d016d468d37b4edbab192d8c94d556bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34063 } } peers { permanent_uuid: "655d707e276145a9a712ab8388e1d5d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38265 } }
I20260927 12:46:43.700752 8438 leader_election.cc:288] T e4940d47723c403cb069dc3ab28c84d7 P d016d468d37b4edbab192d8c94d556bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b5ece2da9d94112a22a017cf744fdd4 (127.0.98.1:37315), 655d707e276145a9a712ab8388e1d5d8 (127.0.98.3:38265)
I20260927 12:46:43.705914 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:43.701189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60711 (local address 127.0.98.3:38265)
0927 12:46:43.701436 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:43.701439 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:43.701568 (+ 129us) server_negotiation.cc:407] Connection header received
0927 12:46:43.701670 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:43.701673 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:43.701732 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:43.701810 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:46:43.702163 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:43.703001 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:43.704606 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:43.705265 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:43.705356 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:43.705418 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:43.705696 (+ 278us) server_negotiation.cc:299] Negotiation successful
0927 12:46:43.705783 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":54,"threads_started":1}
I20260927 12:46:43.705911 8425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:43.701148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43225 (local address 127.0.98.1:37315)
0927 12:46:43.701311 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:43.701315 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:43.701529 (+ 214us) server_negotiation.cc:407] Connection header received
0927 12:46:43.701670 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:43.701673 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:43.701724 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:43.701810 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:46:43.702154 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:43.702940 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:43.704606 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:43.705205 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:43.705347 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:43.705417 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:43.705692 (+ 275us) server_negotiation.cc:299] Negotiation successful
0927 12:46:43.705771 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 12:46:43.706131 8500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:43.701056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37315 (local address 127.0.98.2:43225)
0927 12:46:43.701429 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:43.701445 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:43.701568 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:43.701833 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:43.701836 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:43.701878 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:46:43.702062 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:43.702069 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:43.703068 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:43.703072 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:43.704473 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:43.704483 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:43.705385 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:43.705389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:43.705559 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:43.705593 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 12:46:43.705665 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 12:46:43.705939 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":161,"spinlock_wait_cycles":32896,"thread_start_us":85,"threads_started":1}
I20260927 12:46:43.706146 8501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:43.701112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38265 (local address 127.0.98.2:60711)
0927 12:46:43.701487 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:43.701504 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:43.701570 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:43.701833 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:43.701836 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:43.701880 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:46:43.702035 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:43.702069 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:43.703090 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:43.703093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:43.704473 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:43.704483 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:43.705385 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:43.705389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:43.705534 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:43.705581 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 12:46:43.705661 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 12:46:43.705919 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":115,"spinlock_wait_cycles":8320,"thread_start_us":78,"threads_started":1}
I20260927 12:46:43.706611 8076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4940d47723c403cb069dc3ab28c84d7" candidate_uuid: "d016d468d37b4edbab192d8c94d556bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b5ece2da9d94112a22a017cf744fdd4" is_pre_election: true
I20260927 12:46:43.706610 8356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4940d47723c403cb069dc3ab28c84d7" candidate_uuid: "d016d468d37b4edbab192d8c94d556bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "655d707e276145a9a712ab8388e1d5d8" is_pre_election: true
I20260927 12:46:43.706754 8356 raft_consensus.cc:2454] T e4940d47723c403cb069dc3ab28c84d7 P 655d707e276145a9a712ab8388e1d5d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d016d468d37b4edbab192d8c94d556bd in term 0.
I20260927 12:46:43.706770 8076 raft_consensus.cc:2454] T e4940d47723c403cb069dc3ab28c84d7 P 7b5ece2da9d94112a22a017cf744fdd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d016d468d37b4edbab192d8c94d556bd in term 0.
I20260927 12:46:43.707113 8147 leader_election.cc:302] T e4940d47723c403cb069dc3ab28c84d7 P d016d468d37b4edbab192d8c94d556bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 655d707e276145a9a712ab8388e1d5d8, d016d468d37b4edbab192d8c94d556bd; no voters:
I20260927 12:46:43.707374 8438 raft_consensus.cc:2781] T e4940d47723c403cb069dc3ab28c84d7 P d016d468d37b4edbab192d8c94d556bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:46:43.707434 8438 raft_consensus.cc:482] T e4940d47723c403cb069dc3ab28c84d7 P d016d468d37b4edbab192d8c94d556bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:46:43.707479 8438 raft_consensus.cc:3037] T e4940d47723c403cb069dc3ab28c84d7 P d016d468d37b4edbab192d8c94d556bd [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:46:43.708417 8438 raft_consensus.cc:504] T e4940d47723c403cb069dc3ab28c84d7 P d016d468d37b4edbab192d8c94d556bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5ece2da9d94112a22a017cf744fdd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37315 } } peers { permanent_uuid: "d016d468d37b4edbab192d8c94d556bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34063 } } peers { permanent_uuid: "655d707e276145a9a712ab8388e1d5d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38265 } }
I20260927 12:46:43.708745 8438 leader_election.cc:288] T e4940d47723c403cb069dc3ab28c84d7 P d016d468d37b4edbab192d8c94d556bd [CANDIDATE]: Term 1 election: Requested vote from peers 7b5ece2da9d94112a22a017cf744fdd4 (127.0.98.1:37315), 655d707e276145a9a712ab8388e1d5d8 (127.0.98.3:38265)
I20260927 12:46:43.709112 8356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4940d47723c403cb069dc3ab28c84d7" candidate_uuid: "d016d468d37b4edbab192d8c94d556bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "655d707e276145a9a712ab8388e1d5d8"
I20260927 12:46:43.709138 8076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4940d47723c403cb069dc3ab28c84d7" candidate_uuid: "d016d468d37b4edbab192d8c94d556bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b5ece2da9d94112a22a017cf744fdd4"
I20260927 12:46:43.709226 8356 raft_consensus.cc:3037] T e4940d47723c403cb069dc3ab28c84d7 P 655d707e276145a9a712ab8388e1d5d8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:46:43.709277 8076 raft_consensus.cc:3037] T e4940d47723c403cb069dc3ab28c84d7 P 7b5ece2da9d94112a22a017cf744fdd4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:46:43.710363 8076 raft_consensus.cc:2454] T e4940d47723c403cb069dc3ab28c84d7 P 7b5ece2da9d94112a22a017cf744fdd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d016d468d37b4edbab192d8c94d556bd in term 1.
I20260927 12:46:43.710454 8356 raft_consensus.cc:2454] T e4940d47723c403cb069dc3ab28c84d7 P 655d707e276145a9a712ab8388e1d5d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d016d468d37b4edbab192d8c94d556bd in term 1.
I20260927 12:46:43.710809 8147 leader_election.cc:302] T e4940d47723c403cb069dc3ab28c84d7 P d016d468d37b4edbab192d8c94d556bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 655d707e276145a9a712ab8388e1d5d8, d016d468d37b4edbab192d8c94d556bd; no voters:
I20260927 12:46:43.711019 8438 raft_consensus.cc:2781] T e4940d47723c403cb069dc3ab28c84d7 P d016d468d37b4edbab192d8c94d556bd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:46:43.711207 8438 raft_consensus.cc:686] T e4940d47723c403cb069dc3ab28c84d7 P d016d468d37b4edbab192d8c94d556bd [term 1 LEADER]: Becoming Leader. State: Replica: d016d468d37b4edbab192d8c94d556bd, State: Running, Role: LEADER
I20260927 12:46:43.711458 8438 consensus_queue.cc:237] T e4940d47723c403cb069dc3ab28c84d7 P d016d468d37b4edbab192d8c94d556bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5ece2da9d94112a22a017cf744fdd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37315 } } peers { permanent_uuid: "d016d468d37b4edbab192d8c94d556bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34063 } } peers { permanent_uuid: "655d707e276145a9a712ab8388e1d5d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38265 } }
I20260927 12:46:43.713945 7919 catalog_manager.cc:5495] T e4940d47723c403cb069dc3ab28c84d7 P d016d468d37b4edbab192d8c94d556bd reported cstate change: term changed from 0 to 1, leader changed from <none> to d016d468d37b4edbab192d8c94d556bd (127.0.98.2). New cstate: current_term: 1 leader_uuid: "d016d468d37b4edbab192d8c94d556bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5ece2da9d94112a22a017cf744fdd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d016d468d37b4edbab192d8c94d556bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "655d707e276145a9a712ab8388e1d5d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38265 } health_report { overall_health: UNKNOWN } } }
I20260927 12:46:43.751982 8506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:43.748351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53406 (local address 127.0.98.2:34063)
0927 12:46:43.748576 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:43.748591 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:43.748603 (+ 12us) server_negotiation.cc:407] Connection header received
0927 12:46:43.748693 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:43.748696 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:43.748758 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:43.748873 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:46:43.749117 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:43.749655 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:43.750509 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:43.750739 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:43.750825 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:43.751069 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:46:43.751553 (+ 484us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:46:43.751655 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:43.751736 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 12:46:43.751794 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":64,"threads_started":1}
I20260927 12:46:43.754454 8356 raft_consensus.cc:1261] T e4940d47723c403cb069dc3ab28c84d7 P 655d707e276145a9a712ab8388e1d5d8 [term 1 FOLLOWER]: Refusing update from remote peer d016d468d37b4edbab192d8c94d556bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:46:43.754588 8076 raft_consensus.cc:1261] T e4940d47723c403cb069dc3ab28c84d7 P 7b5ece2da9d94112a22a017cf744fdd4 [term 1 FOLLOWER]: Refusing update from remote peer d016d468d37b4edbab192d8c94d556bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:46:43.754952 8438 consensus_queue.cc:1035] T e4940d47723c403cb069dc3ab28c84d7 P d016d468d37b4edbab192d8c94d556bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "655d707e276145a9a712ab8388e1d5d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:46:43.755155 8503 consensus_queue.cc:1035] T e4940d47723c403cb069dc3ab28c84d7 P d016d468d37b4edbab192d8c94d556bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b5ece2da9d94112a22a017cf744fdd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:46:43.765303 7918 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:46:43.765465 7918 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:46:43.767551 7918 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 27371dbb6cd84fd7be84c8e671ae4f71: Sending DeleteTablet for 3 replicas of tablet e4940d47723c403cb069dc3ab28c84d7
I20260927 12:46:43.768066 8055 tablet_service.cc:1514] Processing DeleteTablet for tablet e4940d47723c403cb069dc3ab28c84d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:46:43 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49210
I20260927 12:46:43.768146 8336 tablet_service.cc:1514] Processing DeleteTablet for tablet e4940d47723c403cb069dc3ab28c84d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:46:43 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45966
I20260927 12:46:43.768265 8454 tablet_replica.cc:330] T e4940d47723c403cb069dc3ab28c84d7 P 7b5ece2da9d94112a22a017cf744fdd4: stopping tablet replica
I20260927 12:46:43.768301 8455 tablet_replica.cc:330] T e4940d47723c403cb069dc3ab28c84d7 P 655d707e276145a9a712ab8388e1d5d8: stopping tablet replica
I20260927 12:46:43.768375 8454 raft_consensus.cc:2229] T e4940d47723c403cb069dc3ab28c84d7 P 7b5ece2da9d94112a22a017cf744fdd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:46:43.768391 8455 raft_consensus.cc:2229] T e4940d47723c403cb069dc3ab28c84d7 P 655d707e276145a9a712ab8388e1d5d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:46:43.768491 8455 raft_consensus.cc:2258] T e4940d47723c403cb069dc3ab28c84d7 P 655d707e276145a9a712ab8388e1d5d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:46:43.768496 8454 raft_consensus.cc:2258] T e4940d47723c403cb069dc3ab28c84d7 P 7b5ece2da9d94112a22a017cf744fdd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:46:43.768663 8196 tablet_service.cc:1514] Processing DeleteTablet for tablet e4940d47723c403cb069dc3ab28c84d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:46:43 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53392
I20260927 12:46:43.768911 8453 tablet_replica.cc:330] T e4940d47723c403cb069dc3ab28c84d7 P d016d468d37b4edbab192d8c94d556bd: stopping tablet replica
I20260927 12:46:43.769008 8453 raft_consensus.cc:2229] T e4940d47723c403cb069dc3ab28c84d7 P d016d468d37b4edbab192d8c94d556bd [term 1 LEADER]: Raft consensus shutting down.
I20260927 12:46:43.769243 8453 raft_consensus.cc:2258] T e4940d47723c403cb069dc3ab28c84d7 P d016d468d37b4edbab192d8c94d556bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:46:43.769315 8455 ts_tablet_manager.cc:1905] T e4940d47723c403cb069dc3ab28c84d7 P 655d707e276145a9a712ab8388e1d5d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:46:43.770006 8454 ts_tablet_manager.cc:1905] T e4940d47723c403cb069dc3ab28c84d7 P 7b5ece2da9d94112a22a017cf744fdd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:46:43.771842 8455 ts_tablet_manager.cc:1918] T e4940d47723c403cb069dc3ab28c84d7 P 655d707e276145a9a712ab8388e1d5d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:46:43.771857 8454 ts_tablet_manager.cc:1918] T e4940d47723c403cb069dc3ab28c84d7 P 7b5ece2da9d94112a22a017cf744fdd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:46:43.771922 8454 log.cc:1196] T e4940d47723c403cb069dc3ab28c84d7 P 7b5ece2da9d94112a22a017cf744fdd4: Deleting WAL directory at /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/ts-0/wal/wals/e4940d47723c403cb069dc3ab28c84d7
I20260927 12:46:43.771934 8455 log.cc:1196] T e4940d47723c403cb069dc3ab28c84d7 P 655d707e276145a9a712ab8388e1d5d8: Deleting WAL directory at /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/ts-2/wal/wals/e4940d47723c403cb069dc3ab28c84d7
I20260927 12:46:43.772150 8454 ts_tablet_manager.cc:1939] T e4940d47723c403cb069dc3ab28c84d7 P 7b5ece2da9d94112a22a017cf744fdd4: Deleting consensus metadata
I20260927 12:46:43.772185 8455 ts_tablet_manager.cc:1939] T e4940d47723c403cb069dc3ab28c84d7 P 655d707e276145a9a712ab8388e1d5d8: Deleting consensus metadata
I20260927 12:46:43.772823 7905 catalog_manager.cc:4841] TS 7b5ece2da9d94112a22a017cf744fdd4 (127.0.98.1:37315): tablet e4940d47723c403cb069dc3ab28c84d7 (table test-table [id=13c3d8f2234643839cdf19c9baf639df]) successfully deleted
I20260927 12:46:43.772848 7903 catalog_manager.cc:4841] TS 655d707e276145a9a712ab8388e1d5d8 (127.0.98.3:38265): tablet e4940d47723c403cb069dc3ab28c84d7 (table test-table [id=13c3d8f2234643839cdf19c9baf639df]) successfully deleted
I20260927 12:46:43.773118 8453 ts_tablet_manager.cc:1905] T e4940d47723c403cb069dc3ab28c84d7 P d016d468d37b4edbab192d8c94d556bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:46:43.773698 7918 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50006:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:46:43.774156 7918 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:46:43.775051 8453 ts_tablet_manager.cc:1918] T e4940d47723c403cb069dc3ab28c84d7 P d016d468d37b4edbab192d8c94d556bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:46:43.775120 8453 log.cc:1196] T e4940d47723c403cb069dc3ab28c84d7 P d016d468d37b4edbab192d8c94d556bd: Deleting WAL directory at /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/ts-1/wal/wals/e4940d47723c403cb069dc3ab28c84d7
I20260927 12:46:43.775352 8453 ts_tablet_manager.cc:1939] T e4940d47723c403cb069dc3ab28c84d7 P d016d468d37b4edbab192d8c94d556bd: Deleting consensus metadata
I20260927 12:46:43.775952 7904 catalog_manager.cc:4841] TS d016d468d37b4edbab192d8c94d556bd (127.0.98.2:34063): tablet e4940d47723c403cb069dc3ab28c84d7 (table test-table [id=13c3d8f2234643839cdf19c9baf639df]) successfully deleted
I20260927 12:46:43.778472 8055 tablet_service.cc:1467] Processing CreateTablet for tablet 8a2ac64b1a9343518850e5350bea28fd (DEFAULT_TABLE table=test-table [id=7d94c484997d4f4a93ff7fc8fe44ce08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:46:43.778721 8055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a2ac64b1a9343518850e5350bea28fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:46:43.778677 8336 tablet_service.cc:1467] Processing CreateTablet for tablet 8a2ac64b1a9343518850e5350bea28fd (DEFAULT_TABLE table=test-table [id=7d94c484997d4f4a93ff7fc8fe44ce08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:46:43.778681 8196 tablet_service.cc:1467] Processing CreateTablet for tablet 8a2ac64b1a9343518850e5350bea28fd (DEFAULT_TABLE table=test-table [id=7d94c484997d4f4a93ff7fc8fe44ce08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:46:43.778910 8336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a2ac64b1a9343518850e5350bea28fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:46:43.778932 8196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a2ac64b1a9343518850e5350bea28fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:46:43.781100 8431 tablet_bootstrap.cc:492] T 8a2ac64b1a9343518850e5350bea28fd P 7b5ece2da9d94112a22a017cf744fdd4: Bootstrap starting.
I20260927 12:46:43.781337 8432 tablet_bootstrap.cc:492] T 8a2ac64b1a9343518850e5350bea28fd P d016d468d37b4edbab192d8c94d556bd: Bootstrap starting.
I20260927 12:46:43.781908 8431 tablet_bootstrap.cc:654] T 8a2ac64b1a9343518850e5350bea28fd P 7b5ece2da9d94112a22a017cf744fdd4: Neither blocks nor log segments found. Creating new log.
I20260927 12:46:43.782196 8432 tablet_bootstrap.cc:654] T 8a2ac64b1a9343518850e5350bea28fd P d016d468d37b4edbab192d8c94d556bd: Neither blocks nor log segments found. Creating new log.
I20260927 12:46:43.782433 8430 tablet_bootstrap.cc:492] T 8a2ac64b1a9343518850e5350bea28fd P 655d707e276145a9a712ab8388e1d5d8: Bootstrap starting.
I20260927 12:46:43.783345 8432 tablet_bootstrap.cc:492] T 8a2ac64b1a9343518850e5350bea28fd P d016d468d37b4edbab192d8c94d556bd: No bootstrap required, opened a new log
I20260927 12:46:43.783356 8430 tablet_bootstrap.cc:654] T 8a2ac64b1a9343518850e5350bea28fd P 655d707e276145a9a712ab8388e1d5d8: Neither blocks nor log segments found. Creating new log.
I20260927 12:46:43.783402 8432 ts_tablet_manager.cc:1397] T 8a2ac64b1a9343518850e5350bea28fd P d016d468d37b4edbab192d8c94d556bd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 12:46:43.783738 8432 raft_consensus.cc:348] T 8a2ac64b1a9343518850e5350bea28fd P d016d468d37b4edbab192d8c94d556bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5ece2da9d94112a22a017cf744fdd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37315 } } peers { permanent_uuid: "d016d468d37b4edbab192d8c94d556bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34063 } } peers { permanent_uuid: "655d707e276145a9a712ab8388e1d5d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38265 } }
I20260927 12:46:43.783842 8432 raft_consensus.cc:374] T 8a2ac64b1a9343518850e5350bea28fd P d016d468d37b4edbab192d8c94d556bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:46:43.783874 8432 raft_consensus.cc:729] T 8a2ac64b1a9343518850e5350bea28fd P d016d468d37b4edbab192d8c94d556bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d016d468d37b4edbab192d8c94d556bd, State: Initialized, Role: FOLLOWER
I20260927 12:46:43.783990 8432 consensus_queue.cc:260] T 8a2ac64b1a9343518850e5350bea28fd P d016d468d37b4edbab192d8c94d556bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5ece2da9d94112a22a017cf744fdd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37315 } } peers { permanent_uuid: "d016d468d37b4edbab192d8c94d556bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34063 } } peers { permanent_uuid: "655d707e276145a9a712ab8388e1d5d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38265 } }
I20260927 12:46:43.784258 8432 ts_tablet_manager.cc:1428] T 8a2ac64b1a9343518850e5350bea28fd P d016d468d37b4edbab192d8c94d556bd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 12:46:43.784566 8430 tablet_bootstrap.cc:492] T 8a2ac64b1a9343518850e5350bea28fd P 655d707e276145a9a712ab8388e1d5d8: No bootstrap required, opened a new log
I20260927 12:46:43.784642 8430 ts_tablet_manager.cc:1397] T 8a2ac64b1a9343518850e5350bea28fd P 655d707e276145a9a712ab8388e1d5d8: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260927 12:46:43.785104 8430 raft_consensus.cc:348] T 8a2ac64b1a9343518850e5350bea28fd P 655d707e276145a9a712ab8388e1d5d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5ece2da9d94112a22a017cf744fdd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37315 } } peers { permanent_uuid: "d016d468d37b4edbab192d8c94d556bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34063 } } peers { permanent_uuid: "655d707e276145a9a712ab8388e1d5d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38265 } }
I20260927 12:46:43.785215 8430 raft_consensus.cc:374] T 8a2ac64b1a9343518850e5350bea28fd P 655d707e276145a9a712ab8388e1d5d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:46:43.785246 8430 raft_consensus.cc:729] T 8a2ac64b1a9343518850e5350bea28fd P 655d707e276145a9a712ab8388e1d5d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 655d707e276145a9a712ab8388e1d5d8, State: Initialized, Role: FOLLOWER
I20260927 12:46:43.785357 8430 consensus_queue.cc:260] T 8a2ac64b1a9343518850e5350bea28fd P 655d707e276145a9a712ab8388e1d5d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5ece2da9d94112a22a017cf744fdd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37315 } } peers { permanent_uuid: "d016d468d37b4edbab192d8c94d556bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34063 } } peers { permanent_uuid: "655d707e276145a9a712ab8388e1d5d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38265 } }
I20260927 12:46:43.785590 8430 ts_tablet_manager.cc:1428] T 8a2ac64b1a9343518850e5350bea28fd P 655d707e276145a9a712ab8388e1d5d8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 12:46:43.786612 8431 tablet_bootstrap.cc:492] T 8a2ac64b1a9343518850e5350bea28fd P 7b5ece2da9d94112a22a017cf744fdd4: No bootstrap required, opened a new log
I20260927 12:46:43.786703 8431 ts_tablet_manager.cc:1397] T 8a2ac64b1a9343518850e5350bea28fd P 7b5ece2da9d94112a22a017cf744fdd4: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260927 12:46:43.787082 8431 raft_consensus.cc:348] T 8a2ac64b1a9343518850e5350bea28fd P 7b5ece2da9d94112a22a017cf744fdd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5ece2da9d94112a22a017cf744fdd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37315 } } peers { permanent_uuid: "d016d468d37b4edbab192d8c94d556bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34063 } } peers { permanent_uuid: "655d707e276145a9a712ab8388e1d5d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38265 } }
I20260927 12:46:43.787180 8431 raft_consensus.cc:374] T 8a2ac64b1a9343518850e5350bea28fd P 7b5ece2da9d94112a22a017cf744fdd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:46:43.787211 8431 raft_consensus.cc:729] T 8a2ac64b1a9343518850e5350bea28fd P 7b5ece2da9d94112a22a017cf744fdd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b5ece2da9d94112a22a017cf744fdd4, State: Initialized, Role: FOLLOWER
I20260927 12:46:43.787326 8431 consensus_queue.cc:260] T 8a2ac64b1a9343518850e5350bea28fd P 7b5ece2da9d94112a22a017cf744fdd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5ece2da9d94112a22a017cf744fdd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37315 } } peers { permanent_uuid: "d016d468d37b4edbab192d8c94d556bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34063 } } peers { permanent_uuid: "655d707e276145a9a712ab8388e1d5d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38265 } }
I20260927 12:46:43.787537 8431 ts_tablet_manager.cc:1428] T 8a2ac64b1a9343518850e5350bea28fd P 7b5ece2da9d94112a22a017cf744fdd4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 12:46:43.864668 8440 raft_consensus.cc:482] T 8a2ac64b1a9343518850e5350bea28fd P 655d707e276145a9a712ab8388e1d5d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:46:43.864835 8440 raft_consensus.cc:504] T 8a2ac64b1a9343518850e5350bea28fd P 655d707e276145a9a712ab8388e1d5d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5ece2da9d94112a22a017cf744fdd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37315 } } peers { permanent_uuid: "d016d468d37b4edbab192d8c94d556bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34063 } } peers { permanent_uuid: "655d707e276145a9a712ab8388e1d5d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38265 } }
I20260927 12:46:43.865331 8440 leader_election.cc:288] T 8a2ac64b1a9343518850e5350bea28fd P 655d707e276145a9a712ab8388e1d5d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b5ece2da9d94112a22a017cf744fdd4 (127.0.98.1:37315), d016d468d37b4edbab192d8c94d556bd (127.0.98.2:34063)
I20260927 12:46:43.865765 8198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ac64b1a9343518850e5350bea28fd" candidate_uuid: "655d707e276145a9a712ab8388e1d5d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d016d468d37b4edbab192d8c94d556bd" is_pre_election: true
I20260927 12:46:43.865809 8076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ac64b1a9343518850e5350bea28fd" candidate_uuid: "655d707e276145a9a712ab8388e1d5d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b5ece2da9d94112a22a017cf744fdd4" is_pre_election: true
I20260927 12:46:43.865916 8198 raft_consensus.cc:2454] T 8a2ac64b1a9343518850e5350bea28fd P d016d468d37b4edbab192d8c94d556bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 655d707e276145a9a712ab8388e1d5d8 in term 0.
I20260927 12:46:43.865964 8076 raft_consensus.cc:2454] T 8a2ac64b1a9343518850e5350bea28fd P 7b5ece2da9d94112a22a017cf744fdd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 655d707e276145a9a712ab8388e1d5d8 in term 0.
I20260927 12:46:43.866297 8289 leader_election.cc:302] T 8a2ac64b1a9343518850e5350bea28fd P 655d707e276145a9a712ab8388e1d5d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 655d707e276145a9a712ab8388e1d5d8, 7b5ece2da9d94112a22a017cf744fdd4, d016d468d37b4edbab192d8c94d556bd; no voters:
I20260927 12:46:43.866468 8440 raft_consensus.cc:2781] T 8a2ac64b1a9343518850e5350bea28fd P 655d707e276145a9a712ab8388e1d5d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:46:43.866528 8440 raft_consensus.cc:482] T 8a2ac64b1a9343518850e5350bea28fd P 655d707e276145a9a712ab8388e1d5d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:46:43.866572 8440 raft_consensus.cc:3037] T 8a2ac64b1a9343518850e5350bea28fd P 655d707e276145a9a712ab8388e1d5d8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:46:43.867573 8440 raft_consensus.cc:504] T 8a2ac64b1a9343518850e5350bea28fd P 655d707e276145a9a712ab8388e1d5d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5ece2da9d94112a22a017cf744fdd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37315 } } peers { permanent_uuid: "d016d468d37b4edbab192d8c94d556bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34063 } } peers { permanent_uuid: "655d707e276145a9a712ab8388e1d5d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38265 } }
I20260927 12:46:43.867933 8440 leader_election.cc:288] T 8a2ac64b1a9343518850e5350bea28fd P 655d707e276145a9a712ab8388e1d5d8 [CANDIDATE]: Term 1 election: Requested vote from peers 7b5ece2da9d94112a22a017cf744fdd4 (127.0.98.1:37315), d016d468d37b4edbab192d8c94d556bd (127.0.98.2:34063)
I20260927 12:46:43.868304 8076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ac64b1a9343518850e5350bea28fd" candidate_uuid: "655d707e276145a9a712ab8388e1d5d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b5ece2da9d94112a22a017cf744fdd4"
I20260927 12:46:43.868428 8076 raft_consensus.cc:3037] T 8a2ac64b1a9343518850e5350bea28fd P 7b5ece2da9d94112a22a017cf744fdd4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:46:43.869626 8076 raft_consensus.cc:2454] T 8a2ac64b1a9343518850e5350bea28fd P 7b5ece2da9d94112a22a017cf744fdd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 655d707e276145a9a712ab8388e1d5d8 in term 1.
I20260927 12:46:43.869905 8198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a2ac64b1a9343518850e5350bea28fd" candidate_uuid: "655d707e276145a9a712ab8388e1d5d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d016d468d37b4edbab192d8c94d556bd"
I20260927 12:46:43.870070 8198 raft_consensus.cc:3037] T 8a2ac64b1a9343518850e5350bea28fd P d016d468d37b4edbab192d8c94d556bd [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:46:43.871273 8198 raft_consensus.cc:2454] T 8a2ac64b1a9343518850e5350bea28fd P d016d468d37b4edbab192d8c94d556bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 655d707e276145a9a712ab8388e1d5d8 in term 1.
I20260927 12:46:43.871603 8288 leader_election.cc:302] T 8a2ac64b1a9343518850e5350bea28fd P 655d707e276145a9a712ab8388e1d5d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 655d707e276145a9a712ab8388e1d5d8, d016d468d37b4edbab192d8c94d556bd; no voters:
I20260927 12:46:43.871794 8440 raft_consensus.cc:2781] T 8a2ac64b1a9343518850e5350bea28fd P 655d707e276145a9a712ab8388e1d5d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:46:43.871953 8440 raft_consensus.cc:686] T 8a2ac64b1a9343518850e5350bea28fd P 655d707e276145a9a712ab8388e1d5d8 [term 1 LEADER]: Becoming Leader. State: Replica: 655d707e276145a9a712ab8388e1d5d8, State: Running, Role: LEADER
I20260927 12:46:43.872102 8440 consensus_queue.cc:237] T 8a2ac64b1a9343518850e5350bea28fd P 655d707e276145a9a712ab8388e1d5d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5ece2da9d94112a22a017cf744fdd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37315 } } peers { permanent_uuid: "d016d468d37b4edbab192d8c94d556bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34063 } } peers { permanent_uuid: "655d707e276145a9a712ab8388e1d5d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38265 } }
I20260927 12:46:43.873819 7919 catalog_manager.cc:5495] T 8a2ac64b1a9343518850e5350bea28fd P 655d707e276145a9a712ab8388e1d5d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 655d707e276145a9a712ab8388e1d5d8 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "655d707e276145a9a712ab8388e1d5d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b5ece2da9d94112a22a017cf744fdd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d016d468d37b4edbab192d8c94d556bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "655d707e276145a9a712ab8388e1d5d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38265 } health_report { overall_health: HEALTHY } } }
I20260927 12:46:43.895488 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:43.892176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45988 (local address 127.0.98.3:38265)
0927 12:46:43.892288 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:43.892292 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:43.892305 (+ 13us) server_negotiation.cc:407] Connection header received
0927 12:46:43.892387 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:43.892390 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:43.892431 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:43.892493 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:46:43.892789 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:43.893289 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:43.894482 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:43.894724 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:43.894803 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:43.895081 (+ 278us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:46:43.895146 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:46:43.895190 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:43.895314 (+ 124us) server_negotiation.cc:299] Negotiation successful
0927 12:46:43.895360 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 12:46:43.906262 8425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:43.902670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41779 (local address 127.0.98.1:37315)
0927 12:46:43.902780 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:43.902784 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:43.902992 (+ 208us) server_negotiation.cc:407] Connection header received
0927 12:46:43.903119 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:43.903121 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:43.903163 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:43.903225 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:46:43.903458 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:43.903945 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:43.905136 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:43.905362 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:43.905442 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:43.905787 (+ 345us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:46:43.905869 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:46:43.905936 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:43.906031 (+ 95us) server_negotiation.cc:299] Negotiation successful
0927 12:46:43.906119 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 12:46:43.906296 8517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:43.902580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37315 (local address 127.0.98.3:41779)
0927 12:46:43.902912 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:43.902928 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:43.903037 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:43.903252 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:43.903254 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:43.903262 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 12:46:43.903370 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:43.903375 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:43.904041 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:43.904044 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:43.905001 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:43.905013 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:43.905471 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:43.905474 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:43.905564 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:43.905621 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 12:46:43.905954 (+ 333us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 12:46:43.905959 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 12:46:43.906019 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 12:46:43.906135 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":84,"threads_started":1}
I20260927 12:46:43.912420 8198 raft_consensus.cc:1261] T 8a2ac64b1a9343518850e5350bea28fd P d016d468d37b4edbab192d8c94d556bd [term 1 FOLLOWER]: Refusing update from remote peer 655d707e276145a9a712ab8388e1d5d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:46:43.912405 8076 raft_consensus.cc:1261] T 8a2ac64b1a9343518850e5350bea28fd P 7b5ece2da9d94112a22a017cf744fdd4 [term 1 FOLLOWER]: Refusing update from remote peer 655d707e276145a9a712ab8388e1d5d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:46:43.912829 8440 consensus_queue.cc:1035] T 8a2ac64b1a9343518850e5350bea28fd P 655d707e276145a9a712ab8388e1d5d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d016d468d37b4edbab192d8c94d556bd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:46:43.913113 8513 consensus_queue.cc:1035] T 8a2ac64b1a9343518850e5350bea28fd P 655d707e276145a9a712ab8388e1d5d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b5ece2da9d94112a22a017cf744fdd4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:46:43.931942 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:43.928548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40775 (local address 127.0.98.3:38265)
0927 12:46:43.928662 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:43.928667 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:43.928795 (+ 128us) server_negotiation.cc:407] Connection header received
0927 12:46:43.928894 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:43.928896 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:43.928937 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:43.928993 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:46:43.929301 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:43.929814 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:43.930900 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:43.931172 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:43.931261 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:43.931499 (+ 238us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:46:43.931577 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:46:43.931635 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:43.931725 (+ 90us) server_negotiation.cc:299] Negotiation successful
0927 12:46:43.931786 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 12:46:43.931942 8522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:43.928475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38265 (local address 127.0.98.1:40775)
0927 12:46:43.928715 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:43.928731 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:43.928827 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:43.929002 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:43.929004 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:43.929010 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 12:46:43.929097 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:43.929101 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:43.929933 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:43.929937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:43.930788 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:43.930798 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:43.931280 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:43.931284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:43.931371 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:43.931397 (+ 26us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 12:46:43.931651 (+ 254us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 12:46:43.931658 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 12:46:43.931715 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 12:46:43.931809 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":80,"threads_started":1}
I20260927 12:46:43.958307 7918 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:46:43.958494 7918 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:46:43.960556 7918 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 27371dbb6cd84fd7be84c8e671ae4f71: Sending DeleteTablet for 3 replicas of tablet 8a2ac64b1a9343518850e5350bea28fd
I20260927 12:46:43.961293 8055 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a2ac64b1a9343518850e5350bea28fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:46:43 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49210
I20260927 12:46:43.961331 8196 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a2ac64b1a9343518850e5350bea28fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:46:43 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53392
I20260927 12:46:43.961336 8336 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a2ac64b1a9343518850e5350bea28fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:46:43 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45966
I20260927 12:46:43.961505 8453 tablet_replica.cc:330] T 8a2ac64b1a9343518850e5350bea28fd P d016d468d37b4edbab192d8c94d556bd: stopping tablet replica
I20260927 12:46:43.961519 8455 tablet_replica.cc:330] T 8a2ac64b1a9343518850e5350bea28fd P 655d707e276145a9a712ab8388e1d5d8: stopping tablet replica
I20260927 12:46:43.961520 8454 tablet_replica.cc:330] T 8a2ac64b1a9343518850e5350bea28fd P 7b5ece2da9d94112a22a017cf744fdd4: stopping tablet replica
I20260927 12:46:43.961614 8453 raft_consensus.cc:2229] T 8a2ac64b1a9343518850e5350bea28fd P d016d468d37b4edbab192d8c94d556bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:46:43.961643 8455 raft_consensus.cc:2229] T 8a2ac64b1a9343518850e5350bea28fd P 655d707e276145a9a712ab8388e1d5d8 [term 1 LEADER]: Raft consensus shutting down.
I20260927 12:46:43.961637 8454 raft_consensus.cc:2229] T 8a2ac64b1a9343518850e5350bea28fd P 7b5ece2da9d94112a22a017cf744fdd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:46:43.961726 8453 raft_consensus.cc:2258] T 8a2ac64b1a9343518850e5350bea28fd P d016d468d37b4edbab192d8c94d556bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:46:43.961756 8454 raft_consensus.cc:2258] T 8a2ac64b1a9343518850e5350bea28fd P 7b5ece2da9d94112a22a017cf744fdd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:46:43.961838 8455 raft_consensus.cc:2258] T 8a2ac64b1a9343518850e5350bea28fd P 655d707e276145a9a712ab8388e1d5d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:46:43.962227 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 7991
I20260927 12:46:43.962427 8453 ts_tablet_manager.cc:1905] T 8a2ac64b1a9343518850e5350bea28fd P d016d468d37b4edbab192d8c94d556bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:46:43.964371 8455 ts_tablet_manager.cc:1905] T 8a2ac64b1a9343518850e5350bea28fd P 655d707e276145a9a712ab8388e1d5d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:46:43.964711 8453 ts_tablet_manager.cc:1918] T 8a2ac64b1a9343518850e5350bea28fd P d016d468d37b4edbab192d8c94d556bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 12:46:43.964779 8453 log.cc:1196] T 8a2ac64b1a9343518850e5350bea28fd P d016d468d37b4edbab192d8c94d556bd: Deleting WAL directory at /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/ts-1/wal/wals/8a2ac64b1a9343518850e5350bea28fd
I20260927 12:46:43.964974 8453 ts_tablet_manager.cc:1939] T 8a2ac64b1a9343518850e5350bea28fd P d016d468d37b4edbab192d8c94d556bd: Deleting consensus metadata
I20260927 12:46:43.965963 7904 catalog_manager.cc:4841] TS d016d468d37b4edbab192d8c94d556bd (127.0.98.2:34063): tablet 8a2ac64b1a9343518850e5350bea28fd (table test-table [id=7d94c484997d4f4a93ff7fc8fe44ce08]) successfully deleted
I20260927 12:46:43.966794 8455 ts_tablet_manager.cc:1918] T 8a2ac64b1a9343518850e5350bea28fd P 655d707e276145a9a712ab8388e1d5d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 12:46:43.966869 8455 log.cc:1196] T 8a2ac64b1a9343518850e5350bea28fd P 655d707e276145a9a712ab8388e1d5d8: Deleting WAL directory at /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790513161160093-392-0/minicluster-data/ts-2/wal/wals/8a2ac64b1a9343518850e5350bea28fd
I20260927 12:46:43.967079 8455 ts_tablet_manager.cc:1939] T 8a2ac64b1a9343518850e5350bea28fd P 655d707e276145a9a712ab8388e1d5d8: Deleting consensus metadata
I20260927 12:46:43.967765 7903 catalog_manager.cc:4841] TS 655d707e276145a9a712ab8388e1d5d8 (127.0.98.3:38265): tablet 8a2ac64b1a9343518850e5350bea28fd (table test-table [id=7d94c484997d4f4a93ff7fc8fe44ce08]) successfully deleted
W20260927 12:46:43.971589 7905 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37315 (error 108)
I20260927 12:46:43.971879 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 8131
I20260927 12:46:43.973132 8526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:43.972734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37315 (local address 127.0.0.1:49238)
0927 12:46:43.972956 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:43.973031 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":61,"threads_started":1}
W20260927 12:46:43.974172 7905 catalog_manager.cc:4584] TS 7b5ece2da9d94112a22a017cf744fdd4 (127.0.98.1:37315): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8a2ac64b1a9343518850e5350bea28fd: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37315: connect: Connection refused (error 111)
I20260927 12:46:43.978199 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 8271
I20260927 12:46:43.984308 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 7887
2026-09-27T12:46:43Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3306 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790513161160093-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:46:44 dist-test-slave-lkn4 krb5kdc[8529](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:46:44 dist-test-slave-lkn4 krb5kdc[8529](info): set up 2 sockets
Sep 27 12:46:44 dist-test-slave-lkn4 krb5kdc[8529](info): commencing operation
krb5kdc: starting...
W20260927 12:46:44.950596 392 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790513161160093-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790513161160093-392-0/krb5kdc/test-user.keytab.
Sep 27 12:46:45 dist-test-slave-lkn4 krb5kdc[8529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:46:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:46:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:46:45.070196 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790513161160093-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790513161160093-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790513161160093-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45337
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45337
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:46:45.178195 8542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:46:45.178478 8542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:46:45.178524 8542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:46:45.182561 8542 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:46:45.182631 8542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:46:45.182657 8542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:46:45.182679 8542 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:46:45.182699 8542 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:46:45.188171 8542 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790513161160093-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790513161160093-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45337
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790513161160093-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:46:45.189257 8542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:46:45.190315 8542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:46:45.198769 8547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:45.198769 8548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:45.198756 8550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:45.199038 8542 server_base.cc:1034] running on GCE node
I20260927 12:46:45.199695 8542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:46:45.200790 8542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:46:45.201967 8542 hybrid_clock.cc:648] HybridClock initialized: now 1790513205201946 us; error 31 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-27T12:46:45Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1223 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:46:45 dist-test-slave-lkn4 krb5kdc[8555](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:46:45 dist-test-slave-lkn4 krb5kdc[8555](info): set up 2 sockets
Sep 27 12:46:45 dist-test-slave-lkn4 krb5kdc[8555](info): commencing operation
krb5kdc: starting...
W20260927 12:46:46.166504 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/krb5kdc/test-user.keytab.
Sep 27 12:46:46 dist-test-slave-lkn4 krb5kdc[8555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:46:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:46:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:46:46.282863 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39093
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39093
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:46:46.388873 8568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:46:46.389242 8568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:46:46.389298 8568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:46:46.393862 8568 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:46:46.393932 8568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:46:46.393950 8568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:46:46.393966 8568 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:46:46.393980 8568 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:46:46.399456 8568 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39093
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:46:46.400558 8568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:46:46.401603 8568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:46:46.410718 8576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:46.410723 8574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:46.410794 8568 server_base.cc:1034] running on GCE node
W20260927 12:46:46.410723 8573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:46.411527 8568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:46:46.412685 8568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:46:46.413864 8568 hybrid_clock.cc:648] HybridClock initialized: now 1790513206413838 us; error 44 us; skew 500 ppm
Sep 27 12:46:46 dist-test-slave-lkn4 krb5kdc[8555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513206, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:46:46.416800 8568 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:46:46.417827 8568 webserver.cc:460] Webserver started at http://127.0.98.62:45727/ using document root <none> and password file <none>
I20260927 12:46:46.418421 8568 fs_manager.cc:359] Metadata directory not provided
I20260927 12:46:46.418488 8568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:46:46.418701 8568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:46:46.420779 8568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/master-0/data/instance:
uuid: "ca88faf7d83d430eb15127b6d7469ff0"
format_stamp: "Formatted at 2026-09-27 12:46:46 on dist-test-slave-lkn4"
server_key: "7c15aaf1ff15e61b74d0502096e628eb"
server_key_iv: "4fec5366e36683e1158b790b3e43942e"
server_key_version: "encryptionkey@0"
I20260927 12:46:46.421357 8568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/master-0/wal/instance:
uuid: "ca88faf7d83d430eb15127b6d7469ff0"
format_stamp: "Formatted at 2026-09-27 12:46:46 on dist-test-slave-lkn4"
server_key: "7c15aaf1ff15e61b74d0502096e628eb"
server_key_iv: "4fec5366e36683e1158b790b3e43942e"
server_key_version: "encryptionkey@0"
I20260927 12:46:46.425215 8568 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 12:46:46.427884 8583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:46.429239 8568 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 12:46:46.429360 8568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/master-0/wal
uuid: "ca88faf7d83d430eb15127b6d7469ff0"
format_stamp: "Formatted at 2026-09-27 12:46:46 on dist-test-slave-lkn4"
server_key: "7c15aaf1ff15e61b74d0502096e628eb"
server_key_iv: "4fec5366e36683e1158b790b3e43942e"
server_key_version: "encryptionkey@0"
I20260927 12:46:46.429461 8568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:46:46.447876 8568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:46:46.451218 8568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:46:46.451406 8568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:46:46.458564 8568 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39093
I20260927 12:46:46.458592 8635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39093 every 8 connection(s)
I20260927 12:46:46.459549 8568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:46:46.462419 8636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:46:46.467970 8636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca88faf7d83d430eb15127b6d7469ff0: Bootstrap starting.
I20260927 12:46:46.468834 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 8568
I20260927 12:46:46.468969 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/master-0/wal/instance
I20260927 12:46:46.469275 392 external_mini_cluster.cc:1428] Setting key 563f80dbd53fcc315efa7a0abccc02c1
I20260927 12:46:46.470911 8636 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca88faf7d83d430eb15127b6d7469ff0: Neither blocks nor log segments found. Creating new log.
I20260927 12:46:46.471624 8636 log.cc:824] T 00000000000000000000000000000000 P ca88faf7d83d430eb15127b6d7469ff0: Log is configured to *not* fsync() on all Append() calls
I20260927 12:46:46.473848 8636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca88faf7d83d430eb15127b6d7469ff0: No bootstrap required, opened a new log
I20260927 12:46:46.478088 8636 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca88faf7d83d430eb15127b6d7469ff0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca88faf7d83d430eb15127b6d7469ff0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39093 } }
I20260927 12:46:46.478323 8636 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca88faf7d83d430eb15127b6d7469ff0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
Sep 27 12:46:46 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:46:46.478361 8636 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca88faf7d83d430eb15127b6d7469ff0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca88faf7d83d430eb15127b6d7469ff0, State: Initialized, Role: FOLLOWER
I20260927 12:46:46.478802 8636 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca88faf7d83d430eb15127b6d7469ff0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca88faf7d83d430eb15127b6d7469ff0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39093 } }
I20260927 12:46:46.478976 8636 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca88faf7d83d430eb15127b6d7469ff0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:46:46.479053 8636 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca88faf7d83d430eb15127b6d7469ff0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:46:46.479152 8636 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca88faf7d83d430eb15127b6d7469ff0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:46:46.480239 8636 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca88faf7d83d430eb15127b6d7469ff0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca88faf7d83d430eb15127b6d7469ff0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39093 } }
I20260927 12:46:46.480594 8636 leader_election.cc:302] T 00000000000000000000000000000000 P ca88faf7d83d430eb15127b6d7469ff0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca88faf7d83d430eb15127b6d7469ff0; no voters:
I20260927 12:46:46.480980 8636 leader_election.cc:288] T 00000000000000000000000000000000 P ca88faf7d83d430eb15127b6d7469ff0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:46:46.481092 8641 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca88faf7d83d430eb15127b6d7469ff0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:46:46.481349 8641 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca88faf7d83d430eb15127b6d7469ff0 [term 1 LEADER]: Becoming Leader. State: Replica: ca88faf7d83d430eb15127b6d7469ff0, State: Running, Role: LEADER
I20260927 12:46:46.481769 8641 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca88faf7d83d430eb15127b6d7469ff0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca88faf7d83d430eb15127b6d7469ff0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39093 } }
I20260927 12:46:46.482415 8636 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca88faf7d83d430eb15127b6d7469ff0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:46:46.483206 8641 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca88faf7d83d430eb15127b6d7469ff0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca88faf7d83d430eb15127b6d7469ff0. Latest consensus state: current_term: 1 leader_uuid: "ca88faf7d83d430eb15127b6d7469ff0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca88faf7d83d430eb15127b6d7469ff0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39093 } } }
I20260927 12:46:46.483311 8641 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca88faf7d83d430eb15127b6d7469ff0 [sys.catalog]: This master's current role is: LEADER
I20260927 12:46:46.483980 8642 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca88faf7d83d430eb15127b6d7469ff0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca88faf7d83d430eb15127b6d7469ff0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca88faf7d83d430eb15127b6d7469ff0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39093 } } }
I20260927 12:46:46.484123 8642 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca88faf7d83d430eb15127b6d7469ff0 [sys.catalog]: This master's current role is: LEADER
I20260927 12:46:46.484269 8639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:46.470446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54234 (local address 127.0.98.62:39093)
0927 12:46:46.470933 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:46.470949 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:46.470995 (+ 46us) server_negotiation.cc:407] Connection header received
0927 12:46:46.471647 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:46.471666 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:46.472071 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:46.472465 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:46.473296 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:46.474389 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:46.475574 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:46.476064 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:46.476177 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:46.478885 (+ 2708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:46.478891 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:46.478907 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:46.478939 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:46.480774 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:46.481281 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:46.481288 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:46.481293 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:46.481440 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:46.481758 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:46.481761 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:46.481763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:46.482305 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:46.482505 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:46.482868 (+ 363us) server_negotiation.cc:299] Negotiation successful
0927 12:46:46.483233 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":255,"thread_start_us":102,"threads_started":1}
I20260927 12:46:46.484426 8648 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:46:46.487990 8648 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:46:46.494163 8648 catalog_manager.cc:1347] Generated new cluster ID: 6f6ba70f3f4d4360bb6a3ce2e95b5ccb
I20260927 12:46:46.494342 8648 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:46:46.507676 8648 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:46:46.508652 8648 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:46:46.520848 8648 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca88faf7d83d430eb15127b6d7469ff0: Generated new TSK 0
I20260927 12:46:46.521788 8648 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 12:46:46.549304 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39093
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:46:46.661080 8662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:46:46.661356 8662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:46:46.661399 8662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:46:46.665478 8662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:46:46.665537 8662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:46:46.665606 8662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:46:46.671182 8662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8662
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:46:46.672237 8662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:46:46.673257 8662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 12:46:46.682569 8662 server_base.cc:1034] running on GCE node
W20260927 12:46:46.682533 8668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:46.682524 8670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:46.682533 8667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:46.683126 8662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:46:46.683939 8662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:46:46.685117 8662 hybrid_clock.cc:648] HybridClock initialized: now 1790513206685076 us; error 59 us; skew 500 ppm
Sep 27 12:46:46 dist-test-slave-lkn4 krb5kdc[8555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513206, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:46:46.689065 8662 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 12:46:46.690204 8662 webserver.cc:460] Webserver started at http://127.0.98.1:33411/ using document root <none> and password file <none>
I20260927 12:46:46.690934 8662 fs_manager.cc:359] Metadata directory not provided
I20260927 12:46:46.691006 8662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:46:46.691215 8662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:46:46.693374 8662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-0/data/instance:
uuid: "743cbd2405964bf5a6c1ddc57e081a76"
format_stamp: "Formatted at 2026-09-27 12:46:46 on dist-test-slave-lkn4"
server_key: "dbf2bfe90d01c87ffe9e3bf1f27c6ad4"
server_key_iv: "81ed098fc9ee51a6fbf47467d66aef31"
server_key_version: "encryptionkey@0"
I20260927 12:46:46.693920 8662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-0/wal/instance:
uuid: "743cbd2405964bf5a6c1ddc57e081a76"
format_stamp: "Formatted at 2026-09-27 12:46:46 on dist-test-slave-lkn4"
server_key: "dbf2bfe90d01c87ffe9e3bf1f27c6ad4"
server_key_iv: "81ed098fc9ee51a6fbf47467d66aef31"
server_key_version: "encryptionkey@0"
I20260927 12:46:46.697827 8662 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260927 12:46:46.700313 8677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:46.701573 8662 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:46:46.701680 8662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-0/wal
uuid: "743cbd2405964bf5a6c1ddc57e081a76"
format_stamp: "Formatted at 2026-09-27 12:46:46 on dist-test-slave-lkn4"
server_key: "dbf2bfe90d01c87ffe9e3bf1f27c6ad4"
server_key_iv: "81ed098fc9ee51a6fbf47467d66aef31"
server_key_version: "encryptionkey@0"
I20260927 12:46:46.701797 8662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:46:46.713923 8662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:46:46.716910 8662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:46:46.717093 8662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:46:46.717694 8662 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:46:46.718613 8662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:46:46.718663 8662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:46.718724 8662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:46:46.718755 8662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:46.727278 8662 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45069
I20260927 12:46:46.727371 8790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45069 every 8 connection(s)
I20260927 12:46:46.728160 8662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:46:46.735751 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 8662
I20260927 12:46:46.735888 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:46:46.736166 392 external_mini_cluster.cc:1428] Setting key f1d895c3272be255d4b411dbd85640fe
Sep 27 12:46:46 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513206, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:46:46.742345 8639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:46.730135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55395 (local address 127.0.98.62:39093)
0927 12:46:46.730285 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:46.730289 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:46.731164 (+ 875us) server_negotiation.cc:407] Connection header received
0927 12:46:46.732642 (+ 1478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:46.732646 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:46.732706 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:46.732781 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:46.734015 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:46.734589 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:46.735830 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:46.736118 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:46.736242 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:46.738789 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:46.738792 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:46.738794 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:46.738818 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:46.740166 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:46.740742 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:46.740746 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:46.740747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:46.740826 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:46.741169 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:46.741171 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:46.741172 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:46.741356 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:46.741455 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:46.742116 (+ 661us) server_negotiation.cc:299] Negotiation successful
0927 12:46:46.742224 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 12:46:46.743512 8793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:46.730542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39093 (local address 127.0.98.1:55395)
0927 12:46:46.731004 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:46.731046 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:46.732428 (+ 1382us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:46.732973 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:46.732980 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:46.733346 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:46.733875 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:46.733889 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:46.734711 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:46.734713 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:46.735692 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:46.735704 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:46.736298 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:46.736303 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:46.736496 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:46.737064 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:46.737080 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:46.738612 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:46.740263 (+ 1651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:46.740270 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:46.740287 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:46.740598 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:46.740907 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:46.740910 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:46.740911 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:46.741080 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:46.741467 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:46.741474 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:46.741820 (+ 346us) client_negotiation.cc:769] Sending connection context
0927 12:46:46.742053 (+ 233us) client_negotiation.cc:240] Negotiation successful
0927 12:46:46.742416 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":116,"threads_started":1}
I20260927 12:46:46.744733 8791 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39093
I20260927 12:46:46.745004 8791 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 12:46:46.745571 8791 heartbeater.cc:511] Master 127.0.98.62:39093 requested a full tablet report, sending...
I20260927 12:46:46.747076 8600 ts_manager.cc:194] Registered new tserver with Master: 743cbd2405964bf5a6c1ddc57e081a76 (127.0.98.1:45069)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 12:46:46.748293 8600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55395
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 12:46:46.763178 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39093
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:46:46.871717 8796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:46:46.871984 8796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:46:46.872027 8796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:46:46.876088 8796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:46:46.876161 8796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:46:46.876227 8796 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:46:46.881736 8796 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8796
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:46:46.882838 8796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:46:46.883846 8796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:46:46.893476 8804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:46.893492 8801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:46.893515 8802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:46.893608 8796 server_base.cc:1034] running on GCE node
I20260927 12:46:46.894277 8796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:46:46.894907 8796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:46:46.896080 8796 hybrid_clock.cc:648] HybridClock initialized: now 1790513206896061 us; error 31 us; skew 500 ppm
Sep 27 12:46:46 dist-test-slave-lkn4 krb5kdc[8555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513206, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:46:46.898936 8796 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 12:46:46.899928 8796 webserver.cc:460] Webserver started at http://127.0.98.2:45679/ using document root <none> and password file <none>
I20260927 12:46:46.900468 8796 fs_manager.cc:359] Metadata directory not provided
I20260927 12:46:46.900534 8796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:46:46.900733 8796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:46:46.902832 8796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-1/data/instance:
uuid: "ba484e87e1fd4131b5f84083b91be7ea"
format_stamp: "Formatted at 2026-09-27 12:46:46 on dist-test-slave-lkn4"
server_key: "8db06be7672ca29f089ba706583a13b4"
server_key_iv: "719740e55c25d79e28cd9f7f10050a93"
server_key_version: "encryptionkey@0"
I20260927 12:46:46.903355 8796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ba484e87e1fd4131b5f84083b91be7ea"
format_stamp: "Formatted at 2026-09-27 12:46:46 on dist-test-slave-lkn4"
server_key: "8db06be7672ca29f089ba706583a13b4"
server_key_iv: "719740e55c25d79e28cd9f7f10050a93"
server_key_version: "encryptionkey@0"
I20260927 12:46:46.907007 8796 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 12:46:46.909737 8811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:46.911175 8796 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 12:46:46.911294 8796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-1/wal
uuid: "ba484e87e1fd4131b5f84083b91be7ea"
format_stamp: "Formatted at 2026-09-27 12:46:46 on dist-test-slave-lkn4"
server_key: "8db06be7672ca29f089ba706583a13b4"
server_key_iv: "719740e55c25d79e28cd9f7f10050a93"
server_key_version: "encryptionkey@0"
I20260927 12:46:46.911407 8796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:46:46.947782 8796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:46:46.951046 8796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:46:46.951225 8796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:46:46.951821 8796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:46:46.952713 8796 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:46:46.952764 8796 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:46.952816 8796 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:46:46.952844 8796 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:46.961282 8796 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45579
I20260927 12:46:46.961315 8924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45579 every 8 connection(s)
I20260927 12:46:46.962239 8796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:46:46.969612 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 8796
I20260927 12:46:46.969749 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:46:46.970069 392 external_mini_cluster.cc:1428] Setting key a79a41cd4d0688b522b18d2c7210399e
Sep 27 12:46:46 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513206, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:46:46.976692 8639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:46.964095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47531 (local address 127.0.98.62:39093)
0927 12:46:46.964218 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:46.964222 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:46.965114 (+ 892us) server_negotiation.cc:407] Connection header received
0927 12:46:46.966571 (+ 1457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:46.966574 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:46.966624 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:46.966701 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:46.967882 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:46.968426 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:46.969577 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:46.969834 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:46.969931 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:46.972814 (+ 2883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:46.972817 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:46.972819 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:46.972843 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:46.974336 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:46.974902 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:46.974905 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:46.974906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:46.974985 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:46.975312 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:46.975314 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:46.975315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:46.975477 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:46.975545 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:46.976340 (+ 795us) server_negotiation.cc:299] Negotiation successful
0927 12:46:46.976526 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 12:46:46.977675 8927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:46.964478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39093 (local address 127.0.98.2:47531)
0927 12:46:46.964945 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:46.964991 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:46.966345 (+ 1354us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:46.966862 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:46.966869 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:46.967231 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:46.967744 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:46.967757 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:46.968533 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:46.968536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:46.969447 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:46.969455 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:46.969974 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:46.969978 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:46.970205 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:46.970782 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:46.970803 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:46.972647 (+ 1844us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:46.974442 (+ 1795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:46.974448 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:46.974466 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:46.974783 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:46.975065 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:46.975068 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:46.975070 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:46.975237 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:46.975624 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:46.975630 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:46.975976 (+ 346us) client_negotiation.cc:769] Sending connection context
0927 12:46:46.976197 (+ 221us) client_negotiation.cc:240] Negotiation successful
0927 12:46:46.976560 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":119,"threads_started":1}
I20260927 12:46:46.978972 8925 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39093
I20260927 12:46:46.979233 8925 heartbeater.cc:464] Registering TS with master...
I20260927 12:46:46.979705 8925 heartbeater.cc:511] Master 127.0.98.62:39093 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 12:46:46.980687 8600 ts_manager.cc:194] Registered new tserver with Master: ba484e87e1fd4131b5f84083b91be7ea (127.0.98.2:45579)
I20260927 12:46:46.981256 8600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47531
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 12:46:46.997253 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39093
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:46:47.109700 8930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:46:47.109972 8930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:46:47.110013 8930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:46:47.114089 8930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:46:47.114164 8930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:46:47.114231 8930 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:46:47.119520 8930 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:46:47.120563 8930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:46:47.121587 8930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:46:47.131255 8938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:47.131254 8936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:47.131301 8930 server_base.cc:1034] running on GCE node
W20260927 12:46:47.131266 8935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:47.131846 8930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:46:47.132506 8930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:46:47.133663 8930 hybrid_clock.cc:648] HybridClock initialized: now 1790513207133640 us; error 44 us; skew 500 ppm
Sep 27 12:46:47 dist-test-slave-lkn4 krb5kdc[8555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513207, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:46:47.136302 8930 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 12:46:47.137095 8930 webserver.cc:460] Webserver started at http://127.0.98.3:39277/ using document root <none> and password file <none>
I20260927 12:46:47.137593 8930 fs_manager.cc:359] Metadata directory not provided
I20260927 12:46:47.137645 8930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:46:47.137807 8930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:46:47.139806 8930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-2/data/instance:
uuid: "a63cc2a4f7d94682ba0a9decaf382d0f"
format_stamp: "Formatted at 2026-09-27 12:46:47 on dist-test-slave-lkn4"
server_key: "7d7022e1c153c18897321c6a5c8da522"
server_key_iv: "2cb09b3e72bcbdafdf8746326897dfb7"
server_key_version: "encryptionkey@0"
I20260927 12:46:47.140300 8930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a63cc2a4f7d94682ba0a9decaf382d0f"
format_stamp: "Formatted at 2026-09-27 12:46:47 on dist-test-slave-lkn4"
server_key: "7d7022e1c153c18897321c6a5c8da522"
server_key_iv: "2cb09b3e72bcbdafdf8746326897dfb7"
server_key_version: "encryptionkey@0"
I20260927 12:46:47.143900 8930 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 12:46:47.146402 8945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:47.147854 8930 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 12:46:47.147987 8930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-2/wal
uuid: "a63cc2a4f7d94682ba0a9decaf382d0f"
format_stamp: "Formatted at 2026-09-27 12:46:47 on dist-test-slave-lkn4"
server_key: "7d7022e1c153c18897321c6a5c8da522"
server_key_iv: "2cb09b3e72bcbdafdf8746326897dfb7"
server_key_version: "encryptionkey@0"
I20260927 12:46:47.148099 8930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:46:47.174319 8930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:46:47.177788 8930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:46:47.177997 8930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:46:47.178666 8930 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:46:47.179546 8930 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:46:47.179594 8930 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:47.179649 8930 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:46:47.179683 8930 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:47.188582 8930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35701
I20260927 12:46:47.188607 9058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35701 every 8 connection(s)
I20260927 12:46:47.189610 8930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:46:47.193723 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 8930
I20260927 12:46:47.193835 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:46:47.194114 392 external_mini_cluster.cc:1428] Setting key 575a08cbeb79eba2bd18364076a78f08
Sep 27 12:46:47 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513207, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:46:47.204128 8639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:47.191530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39663 (local address 127.0.98.62:39093)
0927 12:46:47.191685 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:47.191689 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:47.192734 (+ 1045us) server_negotiation.cc:407] Connection header received
0927 12:46:47.194151 (+ 1417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:47.194156 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:47.194216 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:47.194288 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:47.195610 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:47.196257 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:47.197494 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:47.197758 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:47.197839 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:47.200443 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:47.200447 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:47.200449 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:47.200473 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:47.202000 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:47.202552 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:47.202555 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:47.202556 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:47.202716 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:47.203078 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:47.203081 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:47.203082 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:47.203272 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:47.203360 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:47.203884 (+ 524us) server_negotiation.cc:299] Negotiation successful
0927 12:46:47.204007 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 12:46:47.205092 9061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:47.191997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39093 (local address 127.0.98.3:39663)
0927 12:46:47.192523 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:47.192572 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:47.193915 (+ 1343us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:47.194461 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:47.194469 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:47.194917 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:47.195459 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:47.195479 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:47.196383 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:47.196387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:47.197377 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:47.197386 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:47.197897 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:47.197900 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:47.198103 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:47.198660 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:47.198676 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:47.200259 (+ 1583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:46:47.202183 (+ 1924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:47.202188 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:47.202202 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:47.202461 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:47.202814 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:46:47.202817 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:46:47.202818 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:46:47.202977 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:46:47.203364 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:46:47.203370 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:46:47.203630 (+ 260us) client_negotiation.cc:769] Sending connection context
0927 12:46:47.203795 (+ 165us) client_negotiation.cc:240] Negotiation successful
0927 12:46:47.204064 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":130,"threads_started":1}
I20260927 12:46:47.206441 9059 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39093
I20260927 12:46:47.206768 9059 heartbeater.cc:464] Registering TS with master...
I20260927 12:46:47.207355 9059 heartbeater.cc:511] Master 127.0.98.62:39093 requested a full tablet report, sending...
I20260927 12:46:47.208343 8600 ts_manager.cc:194] Registered new tserver with Master: a63cc2a4f7d94682ba0a9decaf382d0f (127.0.98.3:35701)
I20260927 12:46:47.208874 8600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39663
I20260927 12:46:47.219905 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 12:46:47.288785 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 8662
I20260927 12:46:47.293601 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 8796
I20260927 12:46:47.298020 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 8930
I20260927 12:46:47.302572 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 8568
I20260927 12:46:47.307505 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39093
--webserver_interface=127.0.98.62
--webserver_port=45727
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39093
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:46:47.413448 9068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:46:47.413735 9068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:46:47.413777 9068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:46:47.417773 9068 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:46:47.417846 9068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:46:47.417865 9068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:46:47.417881 9068 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:46:47.417894 9068 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:46:47.423208 9068 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39093
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=45727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9068
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:46:47.424271 9068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:46:47.425313 9068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:46:47.433589 9074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:47.433770 9068 server_base.cc:1034] running on GCE node
W20260927 12:46:47.433694 9076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:47.433595 9073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:47.434485 9068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:46:47.435650 9068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:46:47.436826 9068 hybrid_clock.cc:648] HybridClock initialized: now 1790513207436812 us; error 27 us; skew 500 ppm
Sep 27 12:46:47 dist-test-slave-lkn4 krb5kdc[8555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513207, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:46:47.439502 9068 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:46:47.440421 9068 webserver.cc:460] Webserver started at http://127.0.98.62:45727/ using document root <none> and password file <none>
I20260927 12:46:47.440956 9068 fs_manager.cc:359] Metadata directory not provided
I20260927 12:46:47.441035 9068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:46:47.445072 9068 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 12:46:47.447314 9083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:47.448578 9068 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 12:46:47.448699 9068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/master-0/wal
uuid: "ca88faf7d83d430eb15127b6d7469ff0"
format_stamp: "Formatted at 2026-09-27 12:46:46 on dist-test-slave-lkn4"
server_key: "7c15aaf1ff15e61b74d0502096e628eb"
server_key_iv: "4fec5366e36683e1158b790b3e43942e"
server_key_version: "encryptionkey@0"
I20260927 12:46:47.449285 9068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:46:47.474887 9068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:46:47.477876 9068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:46:47.478088 9068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:46:47.485128 9068 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39093
I20260927 12:46:47.485148 9135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39093 every 8 connection(s)
I20260927 12:46:47.486222 9068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:46:47.490639 9136 sys_catalog.cc:263] Verifying existing consensus state
I20260927 12:46:47.492282 9136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca88faf7d83d430eb15127b6d7469ff0: Bootstrap starting.
I20260927 12:46:47.493466 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 9068
I20260927 12:46:47.493871 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:45069
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:39093
--webserver_port=33411
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 12:46:47.508711 9136 log.cc:824] T 00000000000000000000000000000000 P ca88faf7d83d430eb15127b6d7469ff0: Log is configured to *not* fsync() on all Append() calls
I20260927 12:46:47.515286 9136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca88faf7d83d430eb15127b6d7469ff0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 12:46:47.515604 9136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca88faf7d83d430eb15127b6d7469ff0: Bootstrap complete.
I20260927 12:46:47.519577 9136 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca88faf7d83d430eb15127b6d7469ff0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca88faf7d83d430eb15127b6d7469ff0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39093 } }
I20260927 12:46:47.519976 9136 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca88faf7d83d430eb15127b6d7469ff0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca88faf7d83d430eb15127b6d7469ff0, State: Initialized, Role: FOLLOWER
I20260927 12:46:47.520435 9136 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca88faf7d83d430eb15127b6d7469ff0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca88faf7d83d430eb15127b6d7469ff0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39093 } }
I20260927 12:46:47.520550 9136 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca88faf7d83d430eb15127b6d7469ff0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:46:47.520614 9136 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca88faf7d83d430eb15127b6d7469ff0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:46:47.520706 9136 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca88faf7d83d430eb15127b6d7469ff0 [term 1 FOLLOWER]: Advancing to term 2
I20260927 12:46:47.522632 9136 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca88faf7d83d430eb15127b6d7469ff0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca88faf7d83d430eb15127b6d7469ff0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39093 } }
I20260927 12:46:47.522951 9136 leader_election.cc:302] T 00000000000000000000000000000000 P ca88faf7d83d430eb15127b6d7469ff0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca88faf7d83d430eb15127b6d7469ff0; no voters:
I20260927 12:46:47.523239 9136 leader_election.cc:288] T 00000000000000000000000000000000 P ca88faf7d83d430eb15127b6d7469ff0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 12:46:47.523347 9140 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca88faf7d83d430eb15127b6d7469ff0 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 12:46:47.523527 9140 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca88faf7d83d430eb15127b6d7469ff0 [term 2 LEADER]: Becoming Leader. State: Replica: ca88faf7d83d430eb15127b6d7469ff0, State: Running, Role: LEADER
I20260927 12:46:47.523828 9140 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca88faf7d83d430eb15127b6d7469ff0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca88faf7d83d430eb15127b6d7469ff0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39093 } }
I20260927 12:46:47.524470 9136 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca88faf7d83d430eb15127b6d7469ff0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:46:47.524706 9140 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca88faf7d83d430eb15127b6d7469ff0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca88faf7d83d430eb15127b6d7469ff0. Latest consensus state: current_term: 2 leader_uuid: "ca88faf7d83d430eb15127b6d7469ff0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca88faf7d83d430eb15127b6d7469ff0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39093 } } }
I20260927 12:46:47.524811 9140 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca88faf7d83d430eb15127b6d7469ff0 [sys.catalog]: This master's current role is: LEADER
I20260927 12:46:47.525130 9144 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:46:47.525229 9141 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca88faf7d83d430eb15127b6d7469ff0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ca88faf7d83d430eb15127b6d7469ff0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca88faf7d83d430eb15127b6d7469ff0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39093 } } }
I20260927 12:46:47.525409 9141 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca88faf7d83d430eb15127b6d7469ff0 [sys.catalog]: This master's current role is: LEADER
I20260927 12:46:47.528661 9144 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:46:47.529291 9144 catalog_manager.cc:1259] Loaded cluster ID: 6f6ba70f3f4d4360bb6a3ce2e95b5ccb
I20260927 12:46:47.529342 9144 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:46:47.531064 9144 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:46:47.531584 9144 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ca88faf7d83d430eb15127b6d7469ff0: Loaded TSK: 0
I20260927 12:46:47.532531 9144 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 12:46:47.634826 9138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:46:47.635195 9138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:46:47.635257 9138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:46:47.640632 9138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:46:47.640718 9138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:46:47.640803 9138 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:46:47.646363 9138 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:45069
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=33411
--tserver_master_addrs=127.0.98.62:39093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:46:47.647415 9138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:46:47.648427 9138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:46:47.657987 9162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:47.658003 9163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:47.658183 9165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:47.658620 9138 server_base.cc:1034] running on GCE node
I20260927 12:46:47.659026 9138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:46:47.659716 9138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:46:47.660893 9138 hybrid_clock.cc:648] HybridClock initialized: now 1790513207660870 us; error 37 us; skew 500 ppm
Sep 27 12:46:47 dist-test-slave-lkn4 krb5kdc[8555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513207, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:46:47.663710 9138 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 12:46:47.664646 9138 webserver.cc:460] Webserver started at http://127.0.98.1:33411/ using document root <none> and password file <none>
I20260927 12:46:47.665170 9138 fs_manager.cc:359] Metadata directory not provided
I20260927 12:46:47.665248 9138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:46:47.669118 9138 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:46:47.671257 9172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:47.672451 9138 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 12:46:47.672537 9138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-0/wal
uuid: "743cbd2405964bf5a6c1ddc57e081a76"
format_stamp: "Formatted at 2026-09-27 12:46:46 on dist-test-slave-lkn4"
server_key: "dbf2bfe90d01c87ffe9e3bf1f27c6ad4"
server_key_iv: "81ed098fc9ee51a6fbf47467d66aef31"
server_key_version: "encryptionkey@0"
I20260927 12:46:47.673074 9138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:46:47.694620 9138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:46:47.697748 9138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:46:47.697923 9138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:46:47.698546 9138 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:46:47.699414 9138 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:46:47.699462 9138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:47.699501 9138 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:46:47.699514 9138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:47.707985 9138 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45069
I20260927 12:46:47.708009 9285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45069 every 8 connection(s)
I20260927 12:46:47.708974 9138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:46:47.710332 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 9138
I20260927 12:46:47.710842 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:45579
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:39093
--webserver_port=45679
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 12:46:47 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:46:47 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 12:46:47.722520 9288 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:46:47.711136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54179 (local address 127.0.98.62:39093)
0927 12:46:47.711554 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:47.711569 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:47.711965 (+ 396us) server_negotiation.cc:407] Connection header received
0927 12:46:47.713830 (+ 1865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:47.713841 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:47.714187 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:47.714487 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:47.715699 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:47.716702 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:47.717851 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:47.718346 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:47.718492 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:47.721796 (+ 3304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:54179: BlockingRecv error: recv got EOF from 127.0.98.1:54179 (error 108)
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":105,"threads_started":1}
I20260927 12:46:47.722556 9289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:47.711346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39093 (local address 127.0.98.1:54179)
0927 12:46:47.711802 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:47.711842 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:47.713397 (+ 1555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:47.714589 (+ 1192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:47.714597 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:47.715004 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:47.715456 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:47.715469 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:47.716842 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:47.716846 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:47.717730 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:47.717740 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:47.718492 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:47.718496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:47.718682 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:47.719223 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:47.719239 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:47.721589 (+ 2350us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39093: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":131,"threads_started":1}
W20260927 12:46:47.722975 9286 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39093 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:39093: Client connection negotiation failed: client connection to 127.0.98.62:39093: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 27 12:46:47 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:46:47 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:46:47.728681 9289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:47.723485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39093 (local address 127.0.98.1:60959)
0927 12:46:47.723588 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:47.723602 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:47.723676 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:47.723867 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:47.723869 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:47.724029 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:47.724132 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:47.724138 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:47.724894 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:47.724897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:47.725809 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:47.725820 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:47.726324 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:47.726328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:47.726420 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:47.726844 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:47.726857 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:47.728558 (+ 1701us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39093: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":35}
W20260927 12:46:47.728788 9288 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:46:47.723576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60959 (local address 127.0.98.62:39093)
0927 12:46:47.723675 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:47.723678 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:47.723691 (+ 13us) server_negotiation.cc:407] Connection header received
0927 12:46:47.723740 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:47.723742 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:47.723779 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:47.723833 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:47.724239 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:47.724802 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:47.725957 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:47.726208 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:47.726301 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:47.728691 (+ 2390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:60959: BlockingRecv error: recv got EOF from 127.0.98.1:60959 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 27 12:46:47 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:46:47 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:46:47.734485 9289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:47.729148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39093 (local address 127.0.98.1:54315)
0927 12:46:47.729269 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:47.729282 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:47.729353 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:47.730462 (+ 1109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:47.730465 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:47.730596 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:47.730663 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:47.730668 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:47.731342 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:47.731346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:47.731900 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:47.731908 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:47.732300 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:47.732304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:47.732387 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:47.732797 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:47.732808 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:47.734362 (+ 1554us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39093: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 12:46:47.734567 9288 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:46:47.730154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54315 (local address 127.0.98.62:39093)
0927 12:46:47.730278 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:47.730282 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:47.730293 (+ 11us) server_negotiation.cc:407] Connection header received
0927 12:46:47.730324 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:47.730326 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:47.730362 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:47.730430 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:47.730763 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:47.731250 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:47.732003 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:47.732202 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:47.732289 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:47.734475 (+ 2186us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:54315: BlockingRecv error: recv got EOF from 127.0.98.1:54315 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260927 12:46:47.734889 9286 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 12:46:47.821748 9290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:46:47.822019 9290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:46:47.822084 9290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:46:47.826210 9290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:46:47.826283 9290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:46:47.826349 9290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:46:47.831884 9290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:45579
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=45679
--tserver_master_addrs=127.0.98.62:39093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:46:47.832960 9290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:46:47.833961 9290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:46:47.843652 9296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:47.843675 9295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:47.843652 9298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:47.843819 9290 server_base.cc:1034] running on GCE node
I20260927 12:46:47.844362 9290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:46:47.845005 9290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:46:47.846114 9290 hybrid_clock.cc:648] HybridClock initialized: now 1790513207846083 us; error 37 us; skew 500 ppm
Sep 27 12:46:47 dist-test-slave-lkn4 krb5kdc[8555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513207, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:46:47.848852 9290 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 12:46:47.849740 9290 webserver.cc:460] Webserver started at http://127.0.98.2:45679/ using document root <none> and password file <none>
I20260927 12:46:47.850312 9290 fs_manager.cc:359] Metadata directory not provided
I20260927 12:46:47.850401 9290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:46:47.854519 9290 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 12:46:47.856496 9305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:47.857786 9290 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:46:47.857894 9290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-1/wal
uuid: "ba484e87e1fd4131b5f84083b91be7ea"
format_stamp: "Formatted at 2026-09-27 12:46:46 on dist-test-slave-lkn4"
server_key: "8db06be7672ca29f089ba706583a13b4"
server_key_iv: "719740e55c25d79e28cd9f7f10050a93"
server_key_version: "encryptionkey@0"
I20260927 12:46:47.858481 9290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:46:47.874084 9290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:46:47.876915 9290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:46:47.877097 9290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:46:47.877684 9290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:46:47.878504 9290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:46:47.878551 9290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:47.878587 9290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:46:47.878616 9290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:47.887161 9290 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45579
I20260927 12:46:47.887210 9418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45579 every 8 connection(s)
I20260927 12:46:47.888199 9290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 12:46:47 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:46:47 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:46:47.897746 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 9290
I20260927 12:46:47.898168 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:35701
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:39093
--webserver_port=39277
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:46:47.898825 9288 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:46:47.889892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50573 (local address 127.0.98.62:39093)
0927 12:46:47.890020 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:47.890024 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:47.890861 (+ 837us) server_negotiation.cc:407] Connection header received
0927 12:46:47.892229 (+ 1368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:47.892233 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:47.892290 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:47.892361 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:47.893470 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:47.893983 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:47.895235 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:47.895457 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:47.895548 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:47.898707 (+ 3159us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50573: BlockingRecv error: recv got EOF from 127.0.98.2:50573 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 12:46:47.899310 9421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:47.890272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39093 (local address 127.0.98.2:50573)
0927 12:46:47.890723 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:47.890765 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:47.892026 (+ 1261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:47.892540 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:47.892547 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:47.892889 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:47.893335 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:47.893348 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:47.894108 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:47.894112 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:47.895037 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:47.895046 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:47.895593 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:47.895597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:47.895780 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:47.896284 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:47.896299 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:47.898536 (+ 2237us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39093: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":124,"threads_started":1}
W20260927 12:46:47.899803 9419 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39093 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:39093: Client connection negotiation failed: client connection to 127.0.98.62:39093: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 27 12:46:47 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:46:47 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:46:47.905349 9421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:47.900184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39093 (local address 127.0.98.2:33341)
0927 12:46:47.900303 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:47.900318 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:47.900382 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:47.900571 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:47.900573 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:47.900723 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:47.900826 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:47.900832 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:47.901502 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:47.901505 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:47.902354 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:47.902366 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:47.902807 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:47.902811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:47.902903 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:47.903348 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:47.903364 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:47.905203 (+ 1839us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39093: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260927 12:46:47.905457 9288 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:46:47.900253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33341 (local address 127.0.98.62:39093)
0927 12:46:47.900376 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:47.900381 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:47.900395 (+ 14us) server_negotiation.cc:407] Connection header received
0927 12:46:47.900438 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:47.900441 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:47.900483 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:47.900564 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:47.900925 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:47.901407 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:47.902473 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:47.902693 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:47.902785 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:47.905351 (+ 2566us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:33341: BlockingRecv error: recv got EOF from 127.0.98.2:33341 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 27 12:46:47 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:46:47 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:46:47.911252 9421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:47.906106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39093 (local address 127.0.98.2:55951)
0927 12:46:47.906213 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:47.906228 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:47.906310 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:47.906526 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:47.906529 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:47.906679 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:47.906770 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:47.906777 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:47.907514 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:47.907518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:47.908344 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:47.908355 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:47.908793 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:47.908797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:47.908878 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:47.909321 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:47.909334 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:47.911114 (+ 1780us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39093: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 12:46:47.911327 9288 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:46:47.906171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55951 (local address 127.0.98.62:39093)
0927 12:46:47.906270 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:47.906272 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:47.906284 (+ 12us) server_negotiation.cc:407] Connection header received
0927 12:46:47.906398 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:47.906400 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:47.906437 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:47.906490 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:47.906894 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:47.907393 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:47.908472 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:47.908687 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:47.908768 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:47.911237 (+ 2469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55951: BlockingRecv error: recv got EOF from 127.0.98.2:55951 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260927 12:46:47.911548 9419 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 12:46:48.009357 9422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:46:48.009658 9422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:46:48.009724 9422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:46:48.013816 9422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:46:48.013885 9422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:46:48.013968 9422 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:46:48.019337 9422 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:35701
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=39277
--tserver_master_addrs=127.0.98.62:39093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9422
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:46:48.020383 9422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:46:48.021400 9422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:46:48.032009 9427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:48.032963 9428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:46:48.033033 9430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:46:48.033758 9422 server_base.cc:1034] running on GCE node
I20260927 12:46:48.034271 9422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:46:48.035066 9422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:46:48.036224 9422 hybrid_clock.cc:648] HybridClock initialized: now 1790513208036189 us; error 52 us; skew 500 ppm
Sep 27 12:46:48 dist-test-slave-lkn4 krb5kdc[8555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513208, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:46:48.038985 9422 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 12:46:48.039964 9422 webserver.cc:460] Webserver started at http://127.0.98.3:39277/ using document root <none> and password file <none>
I20260927 12:46:48.040503 9422 fs_manager.cc:359] Metadata directory not provided
I20260927 12:46:48.040609 9422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:46:48.044420 9422 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260927 12:46:48.046663 9437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:48.048038 9422 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:46:48.048146 9422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-2/wal
uuid: "a63cc2a4f7d94682ba0a9decaf382d0f"
format_stamp: "Formatted at 2026-09-27 12:46:47 on dist-test-slave-lkn4"
server_key: "7d7022e1c153c18897321c6a5c8da522"
server_key_iv: "2cb09b3e72bcbdafdf8746326897dfb7"
server_key_version: "encryptionkey@0"
I20260927 12:46:48.048733 9422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:46:48.072129 9422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:46:48.075091 9422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:46:48.075253 9422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:46:48.075863 9422 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:46:48.076674 9422 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:46:48.076719 9422 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:48.076764 9422 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:46:48.076797 9422 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:46:48.085752 9422 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35701
I20260927 12:46:48.085774 9550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35701 every 8 connection(s)
I20260927 12:46:48.086846 9422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:46:48.088995 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 9422
Sep 27 12:46:48 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:46:48 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 12:46:48.098563 9288 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:46:48.088835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56529 (local address 127.0.98.62:39093)
0927 12:46:48.088986 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:48.088991 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:48.089967 (+ 976us) server_negotiation.cc:407] Connection header received
0927 12:46:48.091609 (+ 1642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:48.091612 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:48.091653 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:48.091775 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:48.093073 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:48.093689 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:48.094831 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:48.095103 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:48.095185 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:48.098434 (+ 3249us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56529: BlockingRecv error: recv got EOF from 127.0.98.3:56529 (error 108)
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":54}
I20260927 12:46:48.099004 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:48.089993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54244 (local address 127.0.98.62:39093)
0927 12:46:48.090233 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:48.090237 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:48.090250 (+ 13us) server_negotiation.cc:407] Connection header received
0927 12:46:48.090347 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:48.090350 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:48.090394 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:48.090489 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:48.091341 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:48.092139 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:48.093331 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:48.093551 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:48.093624 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:48.095023 (+ 1399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:46:48.095028 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:46:48.095042 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:46:48.095066 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:46:48.096790 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:48.097266 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:48.097271 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:48.097281 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:48.097388 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:46:48.097682 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:46:48.097686 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:46:48.097687 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:46:48.098165 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:46:48.098353 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 12:46:48.098642 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 12:46:48.098872 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":97,"threads_started":1}
I20260927 12:46:48.099255 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:48.089307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39093 (local address 127.0.98.3:56529)
0927 12:46:48.089819 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:48.089877 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:48.091392 (+ 1515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:48.091965 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:48.091974 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:48.092426 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:48.092928 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:48.092947 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:48.093797 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:48.093800 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:48.094714 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:48.094723 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:48.095260 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:48.095264 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:48.095458 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:48.095854 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:48.095875 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:48.098326 (+ 2451us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39093: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":130,"threads_started":1}
W20260927 12:46:48.099697 9551 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39093 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:39093: Client connection negotiation failed: client connection to 127.0.98.62:39093: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 27 12:46:48 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:46:48 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:46:48.104568 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:48.100078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39093 (local address 127.0.98.3:49461)
0927 12:46:48.100163 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:48.100176 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:48.100248 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:48.100425 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:48.100427 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:48.100538 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:48.100618 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:48.100622 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:48.101400 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:48.101403 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:48.101992 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:48.102000 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:48.102462 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:48.102465 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:48.102536 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:48.102903 (+ 367us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:48.102913 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:48.104478 (+ 1565us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39093: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":26}
W20260927 12:46:48.104838 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:46:48.100147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49461 (local address 127.0.98.62:39093)
0927 12:46:48.100237 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:48.100241 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:48.100253 (+ 12us) server_negotiation.cc:407] Connection header received
0927 12:46:48.100303 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:48.100306 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:48.100341 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:48.100415 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:48.100730 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:48.101308 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:48.102112 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:48.102344 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:48.102419 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:48.104739 (+ 2320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49461: BlockingRecv error: recv got EOF from 127.0.98.3:49461 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 27 12:46:48 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:46:48 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:46:48.109331 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:48.105101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39093 (local address 127.0.98.3:43261)
0927 12:46:48.105212 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:48.105227 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:48.105346 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:48.105524 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:48.105526 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:48.105638 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:48.105701 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:48.105705 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:48.106413 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:48.106415 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:48.106918 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:48.106925 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:48.107306 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:48.107308 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:48.107358 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:48.107740 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:48.107751 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:48.109246 (+ 1495us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39093: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 12:46:48.109426 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:46:48.105164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43261 (local address 127.0.98.62:39093)
0927 12:46:48.105311 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:48.105314 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:48.105325 (+ 11us) server_negotiation.cc:407] Connection header received
0927 12:46:48.105407 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:48.105411 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:48.105446 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:48.105506 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:48.105808 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:48.106336 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:48.107014 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:48.107223 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:48.107299 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:48.109341 (+ 2042us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43261: BlockingRecv error: recv got EOF from 127.0.98.3:43261 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260927 12:46:48.109510 9551 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 12:46:48 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:46:48 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:46:48.741153 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:48.735524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39093 (local address 127.0.98.1:45625)
0927 12:46:48.735889 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:48.735905 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:48.736036 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:48.736397 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:48.736400 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:48.736586 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:48.736719 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:48.736732 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:48.737577 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:48.737580 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:48.738257 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:48.738267 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:48.738742 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:48.738745 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:48.738823 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:48.739345 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:48.739357 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:48.741035 (+ 1678us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39093: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":162,"threads_started":1}
W20260927 12:46:48.741333 9557 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:46:48.735637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45625 (local address 127.0.98.62:39093)
0927 12:46:48.735889 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:48.735894 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:48.736004 (+ 110us) server_negotiation.cc:407] Connection header received
0927 12:46:48.736168 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:48.736172 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:48.736228 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:48.736336 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:48.736847 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:48.737469 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:48.738382 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:48.738604 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:48.738682 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:48.741201 (+ 2519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45625: BlockingRecv error: recv got EOF from 127.0.98.1:45625 (error 108)
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":70,"threads_started":1}
Sep 27 12:46:48 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:46:48 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:46:48.917740 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:48.912186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39093 (local address 127.0.98.2:53649)
0927 12:46:48.912476 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:48.912489 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:48.912598 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:48.912860 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:48.912864 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:48.913064 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:48.913172 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:48.913177 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:48.913900 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:48.913903 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:48.914512 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:48.914521 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:48.914938 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:48.914942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:48.915038 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:48.915519 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:48.915532 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:48.917561 (+ 2029us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39093: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":82,"threads_started":1}
W20260927 12:46:48.917900 9557 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:46:48.912219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53649 (local address 127.0.98.62:39093)
0927 12:46:48.912387 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:48.912391 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:48.912566 (+ 175us) server_negotiation.cc:407] Connection header received
0927 12:46:48.912695 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:48.912698 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:48.912747 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:48.912822 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:48.913281 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:48.913801 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:48.914623 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:48.914844 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:48.914927 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:48.917797 (+ 2870us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53649: BlockingRecv error: recv got EOF from 127.0.98.2:53649 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 27 12:46:49 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:46:49 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:46:49.115904 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:49.110276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39093 (local address 127.0.98.3:47107)
0927 12:46:49.110538 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:49.110553 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:49.110673 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:49.110977 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:49.110980 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:49.111201 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:49.111344 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:49.111353 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:49.112127 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:49.112130 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:49.112701 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:49.112709 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:49.113185 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:49.113187 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:49.113256 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:49.113804 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:49.113815 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:49.115776 (+ 1961us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39093: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":89,"threads_started":1}
W20260927 12:46:49.116006 9557 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:46:49.110403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47107 (local address 127.0.98.62:39093)
0927 12:46:49.110538 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:49.110542 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:49.110626 (+ 84us) server_negotiation.cc:407] Connection header received
0927 12:46:49.110813 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:49.110816 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:49.110865 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:49.110947 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:49.111493 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:49.112022 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:49.112853 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:49.113082 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:49.113159 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:49.115913 (+ 2754us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47107: BlockingRecv error: recv got EOF from 127.0.98.3:47107 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 27 12:46:49 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:46:49 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:46:49.747412 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:49.741970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39093 (local address 127.0.98.1:39983)
0927 12:46:49.742231 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:49.742247 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:49.742344 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:49.742567 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:49.742570 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:49.742741 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:49.742878 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:49.742884 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:49.743602 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:49.743604 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:49.744192 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:49.744221 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:49.744702 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:49.744706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:49.744796 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:49.745371 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:49.745386 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:49.747268 (+ 1882us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39093: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":89,"threads_started":1}
W20260927 12:46:49.747550 9561 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:46:49.742069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39983 (local address 127.0.98.62:39093)
0927 12:46:49.742338 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:49.742341 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:49.742355 (+ 14us) server_negotiation.cc:407] Connection header received
0927 12:46:49.742412 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:49.742414 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:49.742453 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:49.742527 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:49.742985 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:49.743509 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:49.744330 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:49.744596 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:49.744685 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:49.747432 (+ 2747us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:39983: BlockingRecv error: recv got EOF from 127.0.98.1:39983 (error 108)
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":87,"threads_started":1}
Sep 27 12:46:49 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:46:49 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:46:49.923588 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:49.918578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39093 (local address 127.0.98.2:38293)
0927 12:46:49.918869 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:49.918881 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:49.918972 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:49.919164 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:49.919166 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:49.919331 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:49.919431 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:49.919437 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:49.920159 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:49.920161 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:49.920730 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:49.920738 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:49.921121 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:49.921124 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:49.921179 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:49.921632 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:49.921643 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:49.923422 (+ 1779us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39093: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":82,"threads_started":1}
W20260927 12:46:49.923655 9561 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:46:49.918614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38293 (local address 127.0.98.62:39093)
0927 12:46:49.918800 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:49.918806 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:49.918942 (+ 136us) server_negotiation.cc:407] Connection header received
0927 12:46:49.919047 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:49.919049 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:49.919094 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:49.919150 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:49.919510 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:49.920077 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:49.920829 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:49.921040 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:49.921110 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:49.923557 (+ 2447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:38293: BlockingRecv error: recv got EOF from 127.0.98.2:38293 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 27 12:46:50 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:46:50 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:46:50.121997 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:50.116722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39093 (local address 127.0.98.3:33519)
0927 12:46:50.117038 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:50.117054 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:50.117172 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:50.117438 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:50.117440 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:50.117661 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:50.117798 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:50.117807 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:50.118584 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:50.118587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:50.119159 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:50.119167 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:50.119578 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:50.119581 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:50.119654 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:50.120144 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:50.120158 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:50.121837 (+ 1679us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39093: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":90,"threads_started":1}
W20260927 12:46:50.122100 9561 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:46:50.116957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33519 (local address 127.0.98.62:39093)
0927 12:46:50.117069 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:50.117074 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:50.117116 (+ 42us) server_negotiation.cc:407] Connection header received
0927 12:46:50.117287 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:50.117290 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:50.117334 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:50.117400 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:50.117920 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:50.118485 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:50.119271 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:50.119484 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:50.119565 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:50.121977 (+ 2412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33519: BlockingRecv error: recv got EOF from 127.0.98.3:33519 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 27 12:46:50 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:46:50 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:46:50.753773 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:50.748220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39093 (local address 127.0.98.1:42931)
0927 12:46:50.748497 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:50.748515 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:50.748642 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:50.748994 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:50.748997 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:50.749207 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:50.749298 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:50.749304 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:50.750123 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:50.750126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:50.750719 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:50.750728 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:50.751155 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:50.751158 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:50.751239 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:50.751768 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:50.751780 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:50.753599 (+ 1819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39093: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":78,"threads_started":1}
W20260927 12:46:50.753868 9565 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:46:50.748325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42931 (local address 127.0.98.62:39093)
0927 12:46:50.748598 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:50.748602 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:50.748616 (+ 14us) server_negotiation.cc:407] Connection header received
0927 12:46:50.748836 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:50.748839 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:50.748893 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:50.748969 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:50.749414 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:50.749987 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:50.750831 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:50.751057 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:50.751142 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:50.753733 (+ 2591us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42931: BlockingRecv error: recv got EOF from 127.0.98.1:42931 (error 108)
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":91,"threads_started":1}
Sep 27 12:46:50 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:46:50 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:46:50.929816 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:50.924496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39093 (local address 127.0.98.2:37437)
0927 12:46:50.924837 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:50.924852 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:50.924979 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:50.925280 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:50.925283 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:50.925499 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:50.925636 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:50.925646 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:50.926374 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:50.926377 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:50.926949 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:50.926958 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:50.927344 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:50.927346 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:50.927403 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:50.927904 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:50.927917 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:50.929677 (+ 1760us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39093: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":74,"threads_started":1}
W20260927 12:46:50.930030 9565 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:46:50.924560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37437 (local address 127.0.98.62:39093)
0927 12:46:50.924715 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:50.924721 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:50.924940 (+ 219us) server_negotiation.cc:407] Connection header received
0927 12:46:50.925064 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:50.925066 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:50.925118 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:50.925258 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:50.925738 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:50.926278 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:50.927046 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:50.927259 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:50.927330 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:50.929911 (+ 2581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37437: BlockingRecv error: recv got EOF from 127.0.98.2:37437 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 27 12:46:51 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:46:51 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:46:51.132035 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:51.122942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39093 (local address 127.0.98.3:50057)
0927 12:46:51.123205 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:51.123219 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:51.123325 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:51.123547 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:51.123549 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:51.123719 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:51.123870 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:51.123876 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:51.124862 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:51.124866 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:51.125555 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:51.125566 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:51.126076 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:51.126080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:51.126161 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:51.129920 (+ 3759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:51.129934 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:51.131864 (+ 1930us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39093: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":96,"threads_started":1}
W20260927 12:46:51.132141 9565 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:46:51.123021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50057 (local address 127.0.98.62:39093)
0927 12:46:51.123207 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:51.123212 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:51.123307 (+ 95us) server_negotiation.cc:407] Connection header received
0927 12:46:51.123401 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:51.123404 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:51.123452 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:51.123522 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:51.124020 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:51.124761 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:51.125706 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:51.125943 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:51.126062 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:51.132043 (+ 5981us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50057: BlockingRecv error: recv got EOF from 127.0.98.3:50057 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 27 12:46:51 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:46:51 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:46:51.760281 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:51.754554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39093 (local address 127.0.98.1:47615)
0927 12:46:51.754876 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:51.754892 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:51.755019 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:51.755286 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:51.755289 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:51.755497 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:51.755619 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:51.755626 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:51.756548 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:51.756551 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:51.757156 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:51.757165 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:51.757553 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:51.757555 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:51.757621 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:51.758207 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:51.758219 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:51.760155 (+ 1936us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39093: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":82,"threads_started":1}
W20260927 12:46:51.760421 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:46:51.754627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47615 (local address 127.0.98.62:39093)
0927 12:46:51.754898 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:51.754902 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:51.755001 (+ 99us) server_negotiation.cc:407] Connection header received
0927 12:46:51.755108 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:51.755112 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:51.755168 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:51.755244 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:51.755785 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:51.756443 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:51.757256 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:51.757472 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:51.757542 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:51.760292 (+ 2750us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47615: BlockingRecv error: recv got EOF from 127.0.98.1:47615 (error 108)
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":108,"threads_started":1}
Sep 27 12:46:51 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:46:51 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:46:51.936419 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:51.930732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39093 (local address 127.0.98.2:58599)
0927 12:46:51.931016 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:51.931030 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:51.931158 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:51.931423 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:51.931425 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:51.931625 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:51.931716 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:51.931721 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:51.932505 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:51.932509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:51.933152 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:51.933161 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:51.933638 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:51.933641 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:51.933735 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:51.934382 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:51.934394 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:51.936291 (+ 1897us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39093: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":86,"threads_started":1}
W20260927 12:46:51.936556 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:46:51.930812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58599 (local address 127.0.98.62:39093)
0927 12:46:51.930975 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:51.930980 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:51.931111 (+ 131us) server_negotiation.cc:407] Connection header received
0927 12:46:51.931251 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:51.931255 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:51.931306 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:51.931382 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:51.931855 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:51.932394 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:51.933256 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:51.933491 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:51.933593 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:51.936437 (+ 2844us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58599: BlockingRecv error: recv got EOF from 127.0.98.2:58599 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 27 12:46:52 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:46:52 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:46:52.138876 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:52.132886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39093 (local address 127.0.98.3:47099)
0927 12:46:52.133462 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:52.133480 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:52.133605 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:52.133927 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:52.133930 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:52.134181 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:52.134279 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:52.134287 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:52.135040 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:52.135044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:52.135620 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:52.135631 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:52.136078 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:52.136081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:52.136175 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:52.136674 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:52.136686 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:52.138743 (+ 2057us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39093: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":87,"threads_started":1}
W20260927 12:46:52.139010 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:46:52.133004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47099 (local address 127.0.98.62:39093)
0927 12:46:52.133193 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:52.133199 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:52.133548 (+ 349us) server_negotiation.cc:407] Connection header received
0927 12:46:52.133755 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:52.133758 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:52.133807 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:52.133882 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:52.134399 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:52.134931 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:52.135740 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:52.135972 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:52.136066 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:52.138896 (+ 2830us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47099: BlockingRecv error: recv got EOF from 127.0.98.3:47099 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 27 12:46:52 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:46:52 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:46:52.766551 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:52.761028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39093 (local address 127.0.98.1:45151)
0927 12:46:52.761335 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:52.761348 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:52.761436 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:52.761648 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:52.761650 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:52.761885 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:52.761988 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:52.761994 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:52.762808 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:52.762811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:52.763396 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:52.763405 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:52.763923 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:52.763926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:52.763986 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:52.764662 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:52.764674 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:52.766427 (+ 1753us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39093: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":89,"threads_started":1}
W20260927 12:46:52.766772 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:46:52.761122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45151 (local address 127.0.98.62:39093)
0927 12:46:52.761351 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:52.761355 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:52.761395 (+ 40us) server_negotiation.cc:407] Connection header received
0927 12:46:52.761506 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:52.761508 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:52.761549 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:52.761627 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:52.762116 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:52.762641 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:52.763508 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:52.763821 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:52.763911 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:52.766560 (+ 2649us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45151: BlockingRecv error: recv got EOF from 127.0.98.1:45151 (error 108)
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":89,"threads_started":1}
Sep 27 12:46:52 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:46:52 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:46:52.942910 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:52.937343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39093 (local address 127.0.98.2:42151)
0927 12:46:52.937622 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:52.937638 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:52.937813 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:52.938103 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:52.938106 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:52.938329 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:52.938426 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:52.938432 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:52.939174 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:52.939178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:52.939758 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:52.939775 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:52.940199 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:52.940202 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:52.940282 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:52.940780 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:52.940793 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:52.942732 (+ 1939us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39093: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":85,"threads_started":1}
W20260927 12:46:52.942989 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:46:52.937524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42151 (local address 127.0.98.62:39093)
0927 12:46:52.937654 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:52.937659 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:52.937710 (+ 51us) server_negotiation.cc:407] Connection header received
0927 12:46:52.937922 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:52.937925 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:52.937969 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:52.938033 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:52.938551 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:52.939071 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:52.939880 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:52.940097 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:52.940183 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:52.942893 (+ 2710us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42151: BlockingRecv error: recv got EOF from 127.0.98.2:42151 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 27 12:46:53 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:46:53 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:46:53.145336 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:53.139717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39093 (local address 127.0.98.3:48199)
0927 12:46:53.140003 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:53.140019 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:53.140130 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:53.140425 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:53.140428 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:53.140622 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:53.140714 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:53.140719 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:53.141458 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:53.141462 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:53.142028 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:53.142071 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:53.142494 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:53.142498 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:53.142604 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:53.143181 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:53.143193 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:53.145184 (+ 1991us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39093: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":83,"threads_started":1}
W20260927 12:46:53.145435 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:46:53.139903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48199 (local address 127.0.98.62:39093)
0927 12:46:53.140020 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:53.140025 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:53.140076 (+ 51us) server_negotiation.cc:407] Connection header received
0927 12:46:53.140268 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:53.140271 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:53.140320 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:53.140386 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:53.140831 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:53.141357 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:53.142174 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:53.142394 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:53.142480 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:53.145328 (+ 2848us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:48199: BlockingRecv error: recv got EOF from 127.0.98.3:48199 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 27 12:46:53 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:46:53 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:46:53.772974 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:53.767334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39093 (local address 127.0.98.1:49777)
0927 12:46:53.767608 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:53.767624 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:53.767752 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:53.768084 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:53.768087 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:53.768289 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:53.768389 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:53.768396 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:53.769161 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:53.769164 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:53.769911 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:53.769923 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:53.770495 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:53.770498 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:53.770560 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:53.771024 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:53.771036 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:53.772827 (+ 1791us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39093: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":90,"threads_started":1}
W20260927 12:46:53.773068 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:46:53.767432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49777 (local address 127.0.98.62:39093)
0927 12:46:53.767648 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:53.767652 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:53.767703 (+ 51us) server_negotiation.cc:407] Connection header received
0927 12:46:53.767873 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:53.767876 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:53.767926 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:53.768022 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:53.768506 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:53.769053 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:53.770027 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:53.770384 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:53.770478 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:53.772954 (+ 2476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49777: BlockingRecv error: recv got EOF from 127.0.98.1:49777 (error 108)
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":69,"threads_started":1}
Sep 27 12:46:53 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:46:53 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:46:53.949282 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:53.943762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39093 (local address 127.0.98.2:48853)
0927 12:46:53.944035 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:53.944048 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:53.944151 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:53.944433 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:53.944436 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:53.944614 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:53.944720 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:53.944726 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:53.945485 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:53.945488 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:53.946123 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:53.946134 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:53.946660 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:53.946664 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:53.946751 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:53.947360 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:53.947374 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:53.949167 (+ 1793us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39093: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":83,"threads_started":1}
W20260927 12:46:53.949434 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:46:53.943852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48853 (local address 127.0.98.62:39093)
0927 12:46:53.943993 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:53.943997 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:53.944113 (+ 116us) server_negotiation.cc:407] Connection header received
0927 12:46:53.944284 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:53.944286 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:53.944331 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:53.944393 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:53.944855 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:53.945377 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:53.946281 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:53.946539 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:53.946631 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:53.949322 (+ 2691us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48853: BlockingRecv error: recv got EOF from 127.0.98.2:48853 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 27 12:46:54 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:46:54 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:46:54.151190 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:54.146073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39093 (local address 127.0.98.3:33267)
0927 12:46:54.146344 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:54.146359 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:54.146480 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:54.146709 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:54.146712 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:54.146946 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:54.147038 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:54.147044 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:54.147765 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:54.147769 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:54.148319 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:54.148326 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:54.148732 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:54.148743 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:54.148833 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:54.149322 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:54.149340 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:54.151025 (+ 1685us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39093: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":80,"threads_started":1}
W20260927 12:46:54.151264 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:46:54.146174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33267 (local address 127.0.98.62:39093)
0927 12:46:54.146344 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:54.146349 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:54.146436 (+ 87us) server_negotiation.cc:407] Connection header received
0927 12:46:54.146564 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:54.146568 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:54.146613 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:54.146677 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:54.147151 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:54.147662 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:54.148426 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:54.148639 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:54.148723 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:54.151159 (+ 2436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33267: BlockingRecv error: recv got EOF from 127.0.98.3:33267 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Sep 27 12:46:54 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:46:54 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:46:54.779122 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:54.773749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39093 (local address 127.0.98.1:47083)
0927 12:46:54.774110 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:54.774124 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:54.774215 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:54.774489 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:54.774491 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:54.774664 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:54.774762 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:54.774768 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:54.775547 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:54.775549 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:54.776094 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:54.776101 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:54.776586 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:54.776588 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:54.776652 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:54.777108 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:54.777119 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:54.779005 (+ 1886us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39093: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":95,"threads_started":1}
W20260927 12:46:54.779285 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:46:54.773861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47083 (local address 127.0.98.62:39093)
0927 12:46:54.774159 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:54.774163 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:54.774178 (+ 15us) server_negotiation.cc:407] Connection header received
0927 12:46:54.774330 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:54.774333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:54.774382 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:54.774462 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:54.774884 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:54.775457 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:54.776201 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:54.776488 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:54.776570 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:54.779155 (+ 2585us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47083: BlockingRecv error: recv got EOF from 127.0.98.1:47083 (error 108)
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":89,"threads_started":1}
Sep 27 12:46:54 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:46:54 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:46:54.955719 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:54.950074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39093 (local address 127.0.98.2:48679)
0927 12:46:54.950371 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:54.950385 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:54.950488 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:54.950856 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:54.950859 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:54.951034 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:54.951129 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:54.951134 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:54.951901 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:54.951904 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:54.952547 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:54.952557 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:54.953037 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:54.953041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:54.953130 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:54.953670 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:54.953682 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:54.955602 (+ 1920us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39093: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":75,"threads_started":1}
W20260927 12:46:54.955955 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:46:54.950165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48679 (local address 127.0.98.62:39093)
0927 12:46:54.950296 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:54.950300 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:54.950448 (+ 148us) server_negotiation.cc:407] Connection header received
0927 12:46:54.950602 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:54.950605 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:54.950654 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:54.950734 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:54.951233 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:54.951750 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:54.952649 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:54.952917 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:54.953035 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:54.955829 (+ 2794us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48679: BlockingRecv error: recv got EOF from 127.0.98.2:48679 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 27 12:46:55 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:46:55 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:46:55.157428 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:55.152055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39093 (local address 127.0.98.3:34247)
0927 12:46:55.152341 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:55.152355 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:55.152492 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:55.152788 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:55.152790 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:55.152958 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:55.153051 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:55.153056 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:55.153892 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:55.153896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:55.154537 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:55.154547 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:55.154946 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:55.154949 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:55.155008 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:55.155470 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:55.155482 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:55.157314 (+ 1832us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39093: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":77,"threads_started":1}
W20260927 12:46:55.157665 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:46:55.152115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34247 (local address 127.0.98.62:39093)
0927 12:46:55.152279 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:55.152285 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:55.152435 (+ 150us) server_negotiation.cc:407] Connection header received
0927 12:46:55.152570 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:55.152572 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:55.152620 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:55.152689 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:55.153134 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:55.153785 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:55.154631 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:55.154854 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:55.154930 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:55.157483 (+ 2553us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:34247: BlockingRecv error: recv got EOF from 127.0.98.3:34247 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 27 12:46:55 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:46:55 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:46:55.785728 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:55.780002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39093 (local address 127.0.98.1:57929)
0927 12:46:55.780276 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:55.780290 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:55.780380 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:55.780629 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:55.780631 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:55.780812 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:55.780910 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:55.780916 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:55.781725 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:55.781728 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:55.782430 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:55.782442 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:55.782942 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:55.782946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:55.783034 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:55.783687 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:55.783700 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:55.785580 (+ 1880us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39093: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":84,"threads_started":1}
W20260927 12:46:55.785935 9585 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:46:55.780161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57929 (local address 127.0.98.62:39093)
0927 12:46:55.780412 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:55.780416 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:55.780430 (+ 14us) server_negotiation.cc:407] Connection header received
0927 12:46:55.780470 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:55.780472 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:55.780512 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:55.780597 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:55.781028 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:55.781604 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:55.782559 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:55.782823 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:55.782922 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:55.785809 (+ 2887us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:57929: BlockingRecv error: recv got EOF from 127.0.98.1:57929 (error 108)
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":84,"threads_started":1}
Sep 27 12:46:55 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:46:55 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:46:55.962474 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:55.956611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39093 (local address 127.0.98.2:52125)
0927 12:46:55.956941 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:55.956956 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:55.957062 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:55.957300 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:55.957302 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:55.957475 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:55.957572 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:55.957577 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:55.958631 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:55.958634 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:55.959266 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:55.959278 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:55.959772 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:55.959774 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:55.959842 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:55.960309 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:55.960320 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:55.962320 (+ 2000us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39093: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":151,"threads_started":1}
W20260927 12:46:55.962556 9585 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:46:55.956740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52125 (local address 127.0.98.62:39093)
0927 12:46:55.956926 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:55.956929 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:55.957066 (+ 137us) server_negotiation.cc:407] Connection header received
0927 12:46:55.957139 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:55.957142 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:55.957193 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:55.957270 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:55.957703 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:55.958488 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:55.959384 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:55.959609 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:55.959688 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:55.962455 (+ 2767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52125: BlockingRecv error: recv got EOF from 127.0.98.2:52125 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Sep 27 12:46:56 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:46:56 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:46:56.167253 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:56.158288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39093 (local address 127.0.98.3:34937)
0927 12:46:56.158539 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:56.158553 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:56.158659 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:56.158925 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:56.158928 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:56.159104 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:56.159203 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:56.159208 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:56.160052 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:56.160055 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:56.160644 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:56.160653 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:56.161101 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:56.161103 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:56.161164 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:56.165120 (+ 3956us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:56.165134 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:56.167129 (+ 1995us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39093: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":75,"threads_started":1}
W20260927 12:46:56.167419 9585 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:46:56.158378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34937 (local address 127.0.98.62:39093)
0927 12:46:56.158541 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:56.158545 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:56.158608 (+ 63us) server_negotiation.cc:407] Connection header received
0927 12:46:56.158744 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:56.158746 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:56.158794 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:56.158857 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:56.159369 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:56.159927 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:56.160759 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:56.161005 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:56.161089 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:56.167296 (+ 6207us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:34937: BlockingRecv error: recv got EOF from 127.0.98.3:34937 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 27 12:46:56 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:46:56 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:46:56.792178 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:56.786610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39093 (local address 127.0.98.1:43441)
0927 12:46:56.786918 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:56.786932 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:56.787022 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:56.787317 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:56.787319 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:56.787487 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:56.787578 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:56.787583 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:56.788477 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:56.788480 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:56.789092 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:56.789102 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:56.789539 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:56.789542 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:56.789599 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:56.790130 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:56.790142 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:56.792052 (+ 1910us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39093: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":78,"threads_started":1}
W20260927 12:46:56.792353 9589 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:46:56.786716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43441 (local address 127.0.98.62:39093)
0927 12:46:56.786987 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:56.786990 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:56.787005 (+ 15us) server_negotiation.cc:407] Connection header received
0927 12:46:56.787146 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:56.787149 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:56.787204 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:56.787293 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:56.787765 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:56.788359 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:56.789206 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:56.789442 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:56.789531 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:56.792202 (+ 2671us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43441: BlockingRecv error: recv got EOF from 127.0.98.1:43441 (error 108)
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":89,"threads_started":1}
Sep 27 12:46:56 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:46:56 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:46:56.969036 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:56.963290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39093 (local address 127.0.98.2:58265)
0927 12:46:56.963548 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:56.963562 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:56.963722 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:56.964066 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:56.964069 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:56.964265 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:56.964388 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:56.964394 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:56.965241 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:56.965245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:56.965890 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:56.965900 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:56.966396 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:56.966400 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:56.966464 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:56.966979 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:56.966991 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:56.968913 (+ 1922us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39093: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":73,"threads_started":1}
W20260927 12:46:56.969185 9589 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:46:56.963358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58265 (local address 127.0.98.62:39093)
0927 12:46:56.963477 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:56.963481 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:56.963655 (+ 174us) server_negotiation.cc:407] Connection header received
0927 12:46:56.963875 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:56.963878 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:56.963928 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:56.964001 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:56.964529 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:56.965119 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:56.966018 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:56.966269 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:56.966355 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:56.969072 (+ 2717us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58265: BlockingRecv error: recv got EOF from 127.0.98.2:58265 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 27 12:46:57 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:46:57 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:46:57.173339 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:57.167961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39093 (local address 127.0.98.3:36975)
0927 12:46:57.168211 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:57.168224 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:57.168321 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:57.168522 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:57.168524 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:57.168696 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:57.168798 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:57.168804 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:57.169512 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:57.169515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:57.170159 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:57.170169 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:57.170630 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:57.170634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:57.170788 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:57.171246 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:57.171257 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:57.173192 (+ 1935us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39093: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":72,"threads_started":1}
W20260927 12:46:57.173498 9589 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:46:57.168031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36975 (local address 127.0.98.62:39093)
0927 12:46:57.168170 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:57.168175 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:57.168284 (+ 109us) server_negotiation.cc:407] Connection header received
0927 12:46:57.168389 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:57.168391 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:57.168434 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:57.168499 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:57.168888 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:57.169420 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:57.170289 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:57.170511 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:57.170589 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:57.173364 (+ 2775us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:36975: BlockingRecv error: recv got EOF from 127.0.98.3:36975 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 27 12:46:57 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:46:57 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:46:57.798949 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:57.793153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39093 (local address 127.0.98.1:60427)
0927 12:46:57.793435 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:57.793450 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:57.793540 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:57.793897 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:57.793899 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:57.794105 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:57.794232 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:57.794238 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:57.795062 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:57.795065 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:57.795662 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:57.795671 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:57.796148 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:57.796151 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:57.796256 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:57.796872 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:57.796888 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:57.798831 (+ 1943us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39093: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":84,"threads_started":1}
W20260927 12:46:57.799101 9593 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:46:57.793277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60427 (local address 127.0.98.62:39093)
0927 12:46:57.793508 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:57.793511 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:57.793526 (+ 15us) server_negotiation.cc:407] Connection header received
0927 12:46:57.793682 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:57.793685 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:57.793734 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:57.793825 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:57.794358 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:57.794949 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:57.795803 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:57.796042 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:57.796131 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:57.798977 (+ 2846us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:60427: BlockingRecv error: recv got EOF from 127.0.98.1:60427 (error 108)
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":78,"threads_started":1}
Sep 27 12:46:57 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:46:57 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:46:57.975405 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:57.969951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39093 (local address 127.0.98.2:43253)
0927 12:46:57.970255 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:57.970271 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:57.970383 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:57.970614 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:57.970617 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:57.970876 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:57.970978 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:57.970984 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:57.971713 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:57.971716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:57.972285 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:57.972293 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:57.972714 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:57.972718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:57.972798 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:57.973275 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:57.973287 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:57.975293 (+ 2006us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39093: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":105,"threads_started":1}
W20260927 12:46:57.975611 9593 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:46:57.970101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43253 (local address 127.0.98.62:39093)
0927 12:46:57.970258 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:57.970263 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:57.970339 (+ 76us) server_negotiation.cc:407] Connection header received
0927 12:46:57.970467 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:57.970470 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:57.970517 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:57.970583 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:57.971088 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:57.971617 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:57.972396 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:57.972617 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:57.972705 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:57.975491 (+ 2786us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:43253: BlockingRecv error: recv got EOF from 127.0.98.2:43253 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 27 12:46:58 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:46:58 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:46:58.179415 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:58.174194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39093 (local address 127.0.98.3:50887)
0927 12:46:58.174460 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:58.174477 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:58.174585 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:58.174860 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:58.174863 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:58.175047 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:58.175135 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:58.175141 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:58.175863 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:58.175866 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:58.176444 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:58.176452 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:58.176881 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:58.176885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:58.176965 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:58.177494 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:58.177505 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:58.179261 (+ 1756us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39093: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":79,"threads_started":1}
W20260927 12:46:58.179534 9593 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:46:58.174321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50887 (local address 127.0.98.62:39093)
0927 12:46:58.174460 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:58.174465 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:58.174547 (+ 82us) server_negotiation.cc:407] Connection header received
0927 12:46:58.174668 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:58.174671 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:58.174721 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:58.174795 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:58.175254 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:58.175763 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:58.176552 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:58.176776 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:58.176862 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:58.179431 (+ 2569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50887: BlockingRecv error: recv got EOF from 127.0.98.3:50887 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 27 12:46:58 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:46:58 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:46:58.805727 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:58.799769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39093 (local address 127.0.98.1:43871)
0927 12:46:58.800078 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:58.800097 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:58.800242 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:58.800521 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:58.800523 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:58.800730 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:58.800843 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:58.800850 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:58.801694 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:58.801699 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:58.802512 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:58.802524 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:58.803235 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:58.803238 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:58.803308 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:58.803805 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:58.803817 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:58.805578 (+ 1761us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39093: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":87,"threads_started":1}
W20260927 12:46:58.805969 9597 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:46:58.799952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43871 (local address 127.0.98.62:39093)
0927 12:46:58.800191 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:58.800195 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:58.800210 (+ 15us) server_negotiation.cc:407] Connection header received
0927 12:46:58.800360 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:58.800363 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:58.800408 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:58.800485 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:58.800946 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:58.801501 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:58.802644 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:58.803097 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:58.803193 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:58.805720 (+ 2527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43871: BlockingRecv error: recv got EOF from 127.0.98.1:43871 (error 108)
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":84,"threads_started":1}
Sep 27 12:46:58 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:46:58 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:46:58.981957 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:58.976378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39093 (local address 127.0.98.2:45553)
0927 12:46:58.976656 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:58.976670 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:58.976771 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:58.977026 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:58.977029 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:58.977219 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:58.977337 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:58.977342 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:58.978103 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:58.978107 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:58.978818 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:58.978830 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:58.979391 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:58.979396 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:58.979471 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:58.979974 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:58.979986 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:58.981838 (+ 1852us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39093: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":86,"threads_started":1}
W20260927 12:46:58.982095 9597 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:46:58.976516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45553 (local address 127.0.98.62:39093)
0927 12:46:58.976702 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:58.976707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:58.976727 (+ 20us) server_negotiation.cc:407] Connection header received
0927 12:46:58.976855 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:58.976857 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:58.976902 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:58.976981 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:58.977444 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:58.977972 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:58.978931 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:58.979248 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:58.979352 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:58.981970 (+ 2618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45553: BlockingRecv error: recv got EOF from 127.0.98.2:45553 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 27 12:46:59 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:46:59 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:46:59.185930 9599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:59.180225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39093 (local address 127.0.98.3:45645)
0927 12:46:59.180571 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:59.180592 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:59.180703 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:59.181012 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:59.181015 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:59.181245 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:59.181358 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:59.181366 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:59.182141 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:59.182145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:59.182757 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:59.182766 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:59.183235 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:59.183239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:59.183306 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:59.183785 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:59.183797 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:59.185761 (+ 1964us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39093: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":76,"threads_started":1}
W20260927 12:46:59.186023 9597 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:46:59.180284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45645 (local address 127.0.98.62:39093)
0927 12:46:59.180416 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:59.180420 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:59.180670 (+ 250us) server_negotiation.cc:407] Connection header received
0927 12:46:59.180795 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:59.180798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:59.180851 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:59.180932 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:59.181468 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:59.182009 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:59.182871 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:59.183094 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:59.183174 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:59.185911 (+ 2737us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:45645: BlockingRecv error: recv got EOF from 127.0.98.3:45645 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 27 12:46:59 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:46:59 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:46:59.812260 9601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:59.806534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39093 (local address 127.0.98.1:47395)
0927 12:46:59.806931 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:59.806943 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:59.807062 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:59.807342 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:59.807346 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:59.807544 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:59.807652 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:59.807659 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:59.808433 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:59.808437 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:59.809012 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:59.809023 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:59.809484 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:59.809488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:59.809575 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:59.810218 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:59.810231 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:59.812113 (+ 1882us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39093: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":145,"threads_started":1}
W20260927 12:46:59.812390 9600 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:46:59.806561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47395 (local address 127.0.98.62:39093)
0927 12:46:59.806837 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:59.806843 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:59.807018 (+ 175us) server_negotiation.cc:407] Connection header received
0927 12:46:59.807175 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:59.807178 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:59.807226 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:59.807306 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:59.807771 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:59.808319 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:59.809137 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:59.809367 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:59.809461 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:59.812262 (+ 2801us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47395: BlockingRecv error: recv got EOF from 127.0.98.1:47395 (error 108)
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":80,"threads_started":1}
Sep 27 12:46:59 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:46:59 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:46:59.988368 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:46:59.982667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39093 (local address 127.0.98.2:46453)
0927 12:46:59.982959 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0927 12:46:59.982973 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:46:59.983072 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:46:59.983366 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:46:59.983369 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:46:59.983561 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:46:59.983661 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:59.983666 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:59.984446 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:59.984449 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:59.985054 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:46:59.985062 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:46:59.985506 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:46:59.985508 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:46:59.985563 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:59.986096 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:46:59.986108 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:46:59.988251 (+ 2143us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39093: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":105,"threads_started":1}
W20260927 12:46:59.988552 9600 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:46:59.982744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46453 (local address 127.0.98.62:39093)
0927 12:46:59.982899 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 12:46:59.982905 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:46:59.983025 (+ 120us) server_negotiation.cc:407] Connection header received
0927 12:46:59.983148 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:46:59.983150 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:46:59.983192 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:46:59.983252 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:46:59.983815 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:59.984343 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:59.985178 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:46:59.985412 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:46:59.985499 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:46:59.988426 (+ 2927us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46453: BlockingRecv error: recv got EOF from 127.0.98.2:46453 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 27 12:47:00 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:47:00 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:47:00.192163 9603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:47:00.186783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39093 (local address 127.0.98.3:35263)
0927 12:47:00.187060 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0927 12:47:00.187074 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:47:00.187182 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:47:00.187421 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:47:00.187423 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:47:00.187620 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:47:00.187780 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:47:00.187787 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:47:00.188552 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:47:00.188555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:47:00.189109 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:47:00.189116 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:47:00.189542 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:47:00.189545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:47:00.189625 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:47:00.190149 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:47:00.190161 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:47:00.192009 (+ 1848us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39093: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":85,"threads_started":1}
W20260927 12:47:00.192253 9600 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:47:00.186883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35263 (local address 127.0.98.62:39093)
0927 12:47:00.187018 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 12:47:00.187023 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:47:00.187139 (+ 116us) server_negotiation.cc:407] Connection header received
0927 12:47:00.187270 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:47:00.187274 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:47:00.187321 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:47:00.187388 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:47:00.187897 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:47:00.188447 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:47:00.189222 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:47:00.189446 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:47:00.189528 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:47:00.192142 (+ 2614us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:35263: BlockingRecv error: recv got EOF from 127.0.98.3:35263 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 27 12:47:00 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:47:00 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:47:00.818521 9604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:47:00.813157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39093 (local address 127.0.98.1:50799)
0927 12:47:00.813435 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0927 12:47:00.813447 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:47:00.813540 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:47:00.813781 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:47:00.813783 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:47:00.813955 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:47:00.814094 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:47:00.814103 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:47:00.814843 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:47:00.814846 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:47:00.815419 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:47:00.815426 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:47:00.815880 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:47:00.815884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:47:00.815954 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:47:00.816451 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:47:00.816463 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:47:00.818345 (+ 1882us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39093: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":75,"threads_started":1}
W20260927 12:47:00.818598 9605 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:47:00.813279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50799 (local address 127.0.98.62:39093)
0927 12:47:00.813493 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 12:47:00.813497 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:47:00.813511 (+ 14us) server_negotiation.cc:407] Connection header received
0927 12:47:00.813609 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:47:00.813612 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:47:00.813652 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:47:00.813734 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:47:00.814205 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:47:00.814747 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:47:00.815510 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:47:00.815727 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:47:00.815818 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:47:00.818484 (+ 2666us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50799: BlockingRecv error: recv got EOF from 127.0.98.1:50799 (error 108)
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":65,"threads_started":1}
Sep 27 12:47:00 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:47:00 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:47:00.994699 9606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:47:00.989132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39093 (local address 127.0.98.2:34775)
0927 12:47:00.989443 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0927 12:47:00.989458 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:47:00.989574 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:47:00.989868 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:47:00.989872 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:47:00.990101 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:47:00.990201 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:47:00.990207 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:47:00.990933 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:47:00.990937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:47:00.991505 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:47:00.991513 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:47:00.991955 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:47:00.991959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:47:00.992033 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:47:00.992531 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:47:00.992543 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:47:00.994582 (+ 2039us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39093: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":121,"threads_started":1}
W20260927 12:47:00.994817 9605 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:47:00.989293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34775 (local address 127.0.98.62:39093)
0927 12:47:00.989444 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 12:47:00.989449 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:47:00.989520 (+ 71us) server_negotiation.cc:407] Connection header received
0927 12:47:00.989664 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:47:00.989667 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:47:00.989711 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:47:00.989778 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:47:00.990315 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:47:00.990833 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:47:00.991612 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:47:00.991852 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:47:00.991942 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:47:00.994702 (+ 2760us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34775: BlockingRecv error: recv got EOF from 127.0.98.2:34775 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 27 12:47:01 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:47:01 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:47:01.204380 9607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:47:01.193119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39093 (local address 127.0.98.3:46141)
0927 12:47:01.193436 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0927 12:47:01.193452 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:47:01.193602 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:47:01.193877 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:47:01.193880 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:47:01.194119 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:47:01.194229 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:47:01.194235 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:47:01.195003 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:47:01.195006 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:47:01.195731 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:47:01.195742 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:47:01.196294 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:47:01.196298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:47:01.196384 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:47:01.202361 (+ 5977us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:47:01.202376 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:47:01.204256 (+ 1880us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39093: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":110,"threads_started":1}
W20260927 12:47:01.204541 9605 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:47:01.193264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46141 (local address 127.0.98.62:39093)
0927 12:47:01.193437 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 12:47:01.193442 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:47:01.193536 (+ 94us) server_negotiation.cc:407] Connection header received
0927 12:47:01.193701 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:47:01.193704 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:47:01.193764 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:47:01.193834 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:47:01.194342 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:47:01.194894 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:47:01.195853 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:47:01.196161 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:47:01.196249 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:47:01.204420 (+ 8171us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46141: BlockingRecv error: recv got EOF from 127.0.98.3:46141 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 27 12:47:01 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:47:01 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:47:01.824790 9608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:47:01.819344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39093 (local address 127.0.98.1:58279)
0927 12:47:01.819614 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 12:47:01.819630 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:47:01.819744 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:47:01.820007 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:47:01.820009 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:47:01.820180 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:47:01.820287 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:47:01.820294 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:47:01.821096 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:47:01.821098 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:47:01.821668 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:47:01.821677 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:47:01.822109 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:47:01.822112 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:47:01.822173 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:47:01.822718 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:47:01.822733 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:47:01.824622 (+ 1889us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39093: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":72,"threads_started":1}
W20260927 12:47:01.824877 9609 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:47:01.819424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58279 (local address 127.0.98.62:39093)
0927 12:47:01.819717 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0927 12:47:01.819721 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:47:01.819735 (+ 14us) server_negotiation.cc:407] Connection header received
0927 12:47:01.819827 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:47:01.819829 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:47:01.819875 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:47:01.819969 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:47:01.820407 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:47:01.820992 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:47:01.821762 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:47:01.821988 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:47:01.822099 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:47:01.824763 (+ 2664us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58279: BlockingRecv error: recv got EOF from 127.0.98.1:58279 (error 108)
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":75,"threads_started":1}
Sep 27 12:47:01 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:47:01 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:47:02.001288 9610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:47:01.995469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39093 (local address 127.0.98.2:40869)
0927 12:47:01.995719 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0927 12:47:01.995734 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:47:01.995859 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:47:01.996100 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:47:01.996103 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:47:01.996281 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:47:01.996377 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:47:01.996382 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:47:01.997272 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:47:01.997276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:47:01.997919 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:47:01.997929 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:47:01.998430 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:47:01.998434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:47:01.998516 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:47:01.999026 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:47:01.999045 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:47:02.001131 (+ 2086us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39093: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":77,"threads_started":1}
W20260927 12:47:02.001448 9609 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:47:01.995543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40869 (local address 127.0.98.62:39093)
0927 12:47:01.995674 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 12:47:01.995679 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:47:01.995803 (+ 124us) server_negotiation.cc:407] Connection header received
0927 12:47:01.995952 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:47:01.995955 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:47:01.996005 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:47:01.996071 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:47:01.996484 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:47:01.997090 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:47:01.998075 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:47:01.998302 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:47:01.998390 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:47:02.001313 (+ 2923us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:40869: BlockingRecv error: recv got EOF from 127.0.98.2:40869 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 27 12:47:02 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:47:02 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:47:02.210723 9611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:47:02.205258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39093 (local address 127.0.98.3:59015)
0927 12:47:02.205543 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0927 12:47:02.205558 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:47:02.205683 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:47:02.205954 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:47:02.205957 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:47:02.206175 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:47:02.206269 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:47:02.206277 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:47:02.207010 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:47:02.207013 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:47:02.207569 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:47:02.207576 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:47:02.208007 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:47:02.208010 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:47:02.208089 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:47:02.208621 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:47:02.208635 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:47:02.210567 (+ 1932us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39093: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":89,"threads_started":1}
W20260927 12:47:02.211012 9609 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:47:02.205366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59015 (local address 127.0.98.62:39093)
0927 12:47:02.205507 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 12:47:02.205512 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:47:02.205640 (+ 128us) server_negotiation.cc:407] Connection header received
0927 12:47:02.205772 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:47:02.205775 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:47:02.205832 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:47:02.205910 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:47:02.206387 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:47:02.206909 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:47:02.207678 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:47:02.207906 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:47:02.207992 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:47:02.210877 (+ 2885us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59015: BlockingRecv error: recv got EOF from 127.0.98.3:59015 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 27 12:47:02 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:47:02 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:47:02.831260 9612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:47:02.825764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39093 (local address 127.0.98.1:47027)
0927 12:47:02.826025 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0927 12:47:02.826069 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 12:47:02.826156 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:47:02.826415 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:47:02.826418 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:47:02.826649 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:47:02.826803 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:47:02.826810 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:47:02.827558 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:47:02.827561 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:47:02.828183 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:47:02.828193 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:47:02.828623 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:47:02.828625 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:47:02.828686 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:47:02.829233 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:47:02.829246 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:47:02.831095 (+ 1849us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39093: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":68,"threads_started":1}
W20260927 12:47:02.831357 9613 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:47:02.825860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47027 (local address 127.0.98.62:39093)
0927 12:47:02.826133 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0927 12:47:02.826138 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:47:02.826154 (+ 16us) server_negotiation.cc:407] Connection header received
0927 12:47:02.826245 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:47:02.826248 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:47:02.826300 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:47:02.826380 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:47:02.826919 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:47:02.827462 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:47:02.828301 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:47:02.828533 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:47:02.828618 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:47:02.831241 (+ 2623us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47027: BlockingRecv error: recv got EOF from 127.0.98.1:47027 (error 108)
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":86,"threads_started":1}
Sep 27 12:47:03 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:47:03 dist-test-slave-lkn4 krb5kdc[8555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:47:03.007380 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:47:03.002208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39093 (local address 127.0.98.2:58409)
0927 12:47:03.002463 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 12:47:03.002476 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:47:03.002565 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:47:03.002842 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:47:03.002845 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:47:03.003021 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:47:03.003119 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:47:03.003124 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:47:03.003840 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:47:03.003843 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:47:03.004396 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:47:03.004402 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:47:03.004789 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:47:03.004792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:47:03.004847 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:47:03.005340 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:47:03.005351 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:47:03.007214 (+ 1863us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:39093: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":84,"threads_started":1}
W20260927 12:47:03.007462 9613 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:47:03.002305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58409 (local address 127.0.98.62:39093)
0927 12:47:03.002452 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 12:47:03.002456 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:47:03.002526 (+ 70us) server_negotiation.cc:407] Connection header received
0927 12:47:03.002644 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:47:03.002646 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:47:03.002687 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:47:03.002762 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:47:03.003221 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:47:03.003729 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:47:03.004480 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:47:03.004703 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:47:03.004781 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:47:03.007358 (+ 2577us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58409: BlockingRecv error: recv got EOF from 127.0.98.2:58409 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 12:47:03.093621 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 9138
I20260927 12:47:03.099472 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 9290
I20260927 12:47:03.104041 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 9422
I20260927 12:47:03.108839 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 9068
2026-09-27T12:47:03Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17899 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-27T12:47:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:47:03Z Disabled control of system clock
I20260927 12:47:03.137280 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790513161160093-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790513161160093-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790513161160093-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40691
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40691
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 12:47:03.244012 9618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:47:03.244290 9618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:47:03.244324 9618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:47:03.248297 9618 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:47:03.248368 9618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:47:03.248381 9618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:47:03.248397 9618 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:47:03.248411 9618 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:47:03.253225 9618 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790513161160093-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790513161160093-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40691
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9618
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790513161160093-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:47:03.254293 9618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:47:03.255676 9618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:47:03.264283 9626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:47:03.264256 9623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:47:03.264317 9618 server_base.cc:1034] running on GCE node
W20260927 12:47:03.264263 9624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:47:03.265098 9618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:47:03.266286 9618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:47:03.267465 9618 hybrid_clock.cc:648] HybridClock initialized: now 1790513223267446 us; error 37 us; skew 500 ppm
I20260927 12:47:03.269313 9618 webserver.cc:460] Webserver started at http://127.0.98.62:44281/ using document root <none> and password file <none>
I20260927 12:47:03.269855 9618 fs_manager.cc:359] Metadata directory not provided
I20260927 12:47:03.269914 9618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:47:03.270157 9618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:47:03.272213 9618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790513161160093-392-0/minicluster-data/master-0/data/instance:
uuid: "a08e3771b9fe44d98c752f5fe94efd73"
format_stamp: "Formatted at 2026-09-27 12:47:03 on dist-test-slave-lkn4"
server_key: "5de57848d1fc17faeef276fdf98477fa"
server_key_iv: "505766752ea742f5dd2c64e5de55482b"
server_key_version: "encryptionkey@0"
I20260927 12:47:03.272744 9618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790513161160093-392-0/minicluster-data/master-0/wal/instance:
uuid: "a08e3771b9fe44d98c752f5fe94efd73"
format_stamp: "Formatted at 2026-09-27 12:47:03 on dist-test-slave-lkn4"
server_key: "5de57848d1fc17faeef276fdf98477fa"
server_key_iv: "505766752ea742f5dd2c64e5de55482b"
server_key_version: "encryptionkey@0"
I20260927 12:47:03.276448 9618 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 12:47:03.278967 9632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:47:03.280282 9618 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:47:03.280416 9618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790513161160093-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790513161160093-392-0/minicluster-data/master-0/wal
uuid: "a08e3771b9fe44d98c752f5fe94efd73"
format_stamp: "Formatted at 2026-09-27 12:47:03 on dist-test-slave-lkn4"
server_key: "5de57848d1fc17faeef276fdf98477fa"
server_key_iv: "505766752ea742f5dd2c64e5de55482b"
server_key_version: "encryptionkey@0"
I20260927 12:47:03.280531 9618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790513161160093-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790513161160093-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790513161160093-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:47:03.292354 9618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:47:03.293066 9618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:47:03.293257 9618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:47:03.300421 9618 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40691
I20260927 12:47:03.300447 9684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40691 every 8 connection(s)
I20260927 12:47:03.301458 9618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:47:03.302611 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 9618
I20260927 12:47:03.302729 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790513161160093-392-0/minicluster-data/master-0/wal/instance
I20260927 12:47:03.303040 392 external_mini_cluster.cc:1428] Setting key 77cf5262fbd63dd0c4d85cd7d3ae5dd0
I20260927 12:47:03.305158 9685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:47:03.311852 9685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a08e3771b9fe44d98c752f5fe94efd73: Bootstrap starting.
I20260927 12:47:03.313381 9688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:47:03.304391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48684 (local address 127.0.98.62:40691)
0927 12:47:03.305059 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0927 12:47:03.305080 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 12:47:03.305122 (+ 42us) server_negotiation.cc:407] Connection header received
0927 12:47:03.305959 (+ 837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:47:03.305981 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:47:03.306534 (+ 553us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:47:03.306893 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:47:03.307320 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:47:03.308780 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:47:03.309684 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:47:03.310394 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:47:03.310552 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:47:03.311291 (+ 739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:47:03.311297 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:47:03.311314 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:47:03.311807 (+ 493us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:47:03.311877 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:47:03.311903 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:47:03.311995 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 12:47:03.312163 (+ 168us) server_negotiation.cc:299] Negotiation successful
0927 12:47:03.312452 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":119,"threads_started":1}
I20260927 12:47:03.314540 9685 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a08e3771b9fe44d98c752f5fe94efd73: Neither blocks nor log segments found. Creating new log.
I20260927 12:47:03.315275 9685 log.cc:824] T 00000000000000000000000000000000 P a08e3771b9fe44d98c752f5fe94efd73: Log is configured to *not* fsync() on all Append() calls
I20260927 12:47:03.317499 9685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a08e3771b9fe44d98c752f5fe94efd73: No bootstrap required, opened a new log
I20260927 12:47:03.321228 9685 raft_consensus.cc:348] T 00000000000000000000000000000000 P a08e3771b9fe44d98c752f5fe94efd73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a08e3771b9fe44d98c752f5fe94efd73" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40691 } }
I20260927 12:47:03.321556 9685 raft_consensus.cc:374] T 00000000000000000000000000000000 P a08e3771b9fe44d98c752f5fe94efd73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:47:03.321622 9685 raft_consensus.cc:729] T 00000000000000000000000000000000 P a08e3771b9fe44d98c752f5fe94efd73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a08e3771b9fe44d98c752f5fe94efd73, State: Initialized, Role: FOLLOWER
I20260927 12:47:03.322127 9685 consensus_queue.cc:260] T 00000000000000000000000000000000 P a08e3771b9fe44d98c752f5fe94efd73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a08e3771b9fe44d98c752f5fe94efd73" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40691 } }
I20260927 12:47:03.322259 9685 raft_consensus.cc:388] T 00000000000000000000000000000000 P a08e3771b9fe44d98c752f5fe94efd73 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:47:03.322324 9685 raft_consensus.cc:482] T 00000000000000000000000000000000 P a08e3771b9fe44d98c752f5fe94efd73 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:47:03.322417 9685 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a08e3771b9fe44d98c752f5fe94efd73 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:47:03.323375 9685 raft_consensus.cc:504] T 00000000000000000000000000000000 P a08e3771b9fe44d98c752f5fe94efd73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a08e3771b9fe44d98c752f5fe94efd73" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40691 } }
I20260927 12:47:03.323707 9685 leader_election.cc:302] T 00000000000000000000000000000000 P a08e3771b9fe44d98c752f5fe94efd73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a08e3771b9fe44d98c752f5fe94efd73; no voters:
I20260927 12:47:03.324010 9685 leader_election.cc:288] T 00000000000000000000000000000000 P a08e3771b9fe44d98c752f5fe94efd73 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:47:03.324154 9690 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a08e3771b9fe44d98c752f5fe94efd73 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:47:03.324370 9690 raft_consensus.cc:686] T 00000000000000000000000000000000 P a08e3771b9fe44d98c752f5fe94efd73 [term 1 LEADER]: Becoming Leader. State: Replica: a08e3771b9fe44d98c752f5fe94efd73, State: Running, Role: LEADER
I20260927 12:47:03.324663 9690 consensus_queue.cc:237] T 00000000000000000000000000000000 P a08e3771b9fe44d98c752f5fe94efd73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a08e3771b9fe44d98c752f5fe94efd73" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40691 } }
I20260927 12:47:03.325189 9685 sys_catalog.cc:564] T 00000000000000000000000000000000 P a08e3771b9fe44d98c752f5fe94efd73 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:47:03.326280 9692 sys_catalog.cc:455] T 00000000000000000000000000000000 P a08e3771b9fe44d98c752f5fe94efd73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a08e3771b9fe44d98c752f5fe94efd73. Latest consensus state: current_term: 1 leader_uuid: "a08e3771b9fe44d98c752f5fe94efd73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a08e3771b9fe44d98c752f5fe94efd73" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40691 } } }
I20260927 12:47:03.326400 9692 sys_catalog.cc:458] T 00000000000000000000000000000000 P a08e3771b9fe44d98c752f5fe94efd73 [sys.catalog]: This master's current role is: LEADER
I20260927 12:47:03.326711 9691 sys_catalog.cc:455] T 00000000000000000000000000000000 P a08e3771b9fe44d98c752f5fe94efd73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a08e3771b9fe44d98c752f5fe94efd73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a08e3771b9fe44d98c752f5fe94efd73" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40691 } } }
I20260927 12:47:03.326761 9699 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:47:03.326846 9691 sys_catalog.cc:458] T 00000000000000000000000000000000 P a08e3771b9fe44d98c752f5fe94efd73 [sys.catalog]: This master's current role is: LEADER
I20260927 12:47:03.329712 9699 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:47:03.335278 9699 catalog_manager.cc:1347] Generated new cluster ID: 34d0cde16ee04e1b91275ba6a0d4098a
I20260927 12:47:03.335347 9699 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:47:03.347352 9699 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:47:03.348353 9699 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:47:03.352105 9699 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a08e3771b9fe44d98c752f5fe94efd73: Generated new TSK 0
I20260927 12:47:03.352857 9699 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:47:03.364454 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790513161160093-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790513161160093-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790513161160093-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40691
--builtin_ntp_servers=127.0.98.20:35397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 12:47:03.476862 9709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:47:03.477155 9709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:47:03.477193 9709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:47:03.481194 9709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:47:03.481254 9709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:47:03.481307 9709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:47:03.486181 9709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790513161160093-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790513161160093-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790513161160093-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:47:03.487290 9709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:47:03.488725 9709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:47:03.498333 9714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:47:03.498350 9715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:47:03.498333 9717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:47:03.498553 9709 server_base.cc:1034] running on GCE node
I20260927 12:47:03.498939 9709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:47:03.499560 9709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:47:03.500735 9709 hybrid_clock.cc:648] HybridClock initialized: now 1790513223500708 us; error 39 us; skew 500 ppm
I20260927 12:47:03.502660 9709 webserver.cc:460] Webserver started at http://127.0.98.1:39611/ using document root <none> and password file <none>
I20260927 12:47:03.503213 9709 fs_manager.cc:359] Metadata directory not provided
I20260927 12:47:03.503281 9709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:47:03.503477 9709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:47:03.505484 9709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790513161160093-392-0/minicluster-data/ts-0/data/instance:
uuid: "02454500a8094200b291e5edee301b7b"
format_stamp: "Formatted at 2026-09-27 12:47:03 on dist-test-slave-lkn4"
server_key: "fc3c1cd436fe24368750ff41cdcf5301"
server_key_iv: "002aba548740608203bb7fa7e1cb1076"
server_key_version: "encryptionkey@0"
I20260927 12:47:03.506019 9709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790513161160093-392-0/minicluster-data/ts-0/wal/instance:
uuid: "02454500a8094200b291e5edee301b7b"
format_stamp: "Formatted at 2026-09-27 12:47:03 on dist-test-slave-lkn4"
server_key: "fc3c1cd436fe24368750ff41cdcf5301"
server_key_iv: "002aba548740608203bb7fa7e1cb1076"
server_key_version: "encryptionkey@0"
I20260927 12:47:03.509800 9709 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:47:03.512408 9723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:47:03.513604 9709 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 12:47:03.513712 9709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790513161160093-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790513161160093-392-0/minicluster-data/ts-0/wal
uuid: "02454500a8094200b291e5edee301b7b"
format_stamp: "Formatted at 2026-09-27 12:47:03 on dist-test-slave-lkn4"
server_key: "fc3c1cd436fe24368750ff41cdcf5301"
server_key_iv: "002aba548740608203bb7fa7e1cb1076"
server_key_version: "encryptionkey@0"
I20260927 12:47:03.513818 9709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790513161160093-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790513161160093-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790513161160093-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:47:03.533291 9709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:47:03.533987 9709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:47:03.534194 9709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:47:03.534842 9709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:47:03.535774 9709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:47:03.535821 9709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:47:03.535872 9709 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:47:03.535902 9709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:47:03.544260 9709 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43431
I20260927 12:47:03.544284 9836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43431 every 8 connection(s)
I20260927 12:47:03.545168 9709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:47:03.550972 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 9709
I20260927 12:47:03.551090 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790513161160093-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:47:03.551393 392 external_mini_cluster.cc:1428] Setting key d61636fe1cd40e1cad7ad56be7e5792b
I20260927 12:47:03.553496 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790513161160093-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790513161160093-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790513161160093-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40691
--builtin_ntp_servers=127.0.98.20:35397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 12:47:03.554945 9688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:47:03.547002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42895 (local address 127.0.98.62:40691)
0927 12:47:03.547153 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 12:47:03.547157 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:47:03.548197 (+ 1040us) server_negotiation.cc:407] Connection header received
0927 12:47:03.549615 (+ 1418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:47:03.549619 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:47:03.549669 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:47:03.549740 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:47:03.550719 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:47:03.551496 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:47:03.553004 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:47:03.553303 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:47:03.553414 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:47:03.554489 (+ 1075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:47:03.554492 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:47:03.554495 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:47:03.554508 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:47:03.554531 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:47:03.554541 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:47:03.554609 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 12:47:03.554770 (+ 161us) server_negotiation.cc:299] Negotiation successful
0927 12:47:03.554838 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 12:47:03.556049 9839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:47:03.547382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40691 (local address 127.0.98.1:42895)
0927 12:47:03.548059 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0927 12:47:03.548101 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 12:47:03.549423 (+ 1322us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:47:03.549934 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:47:03.549942 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:47:03.550032 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:47:03.550570 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:47:03.550583 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:47:03.551612 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:47:03.551615 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:47:03.552880 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:47:03.552892 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:47:03.553437 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:47:03.553440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:47:03.553590 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:47:03.554233 (+ 643us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 12:47:03.554235 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:47:03.554282 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 12:47:03.554348 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:47:03.554629 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:47:03.554637 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:47:03.554643 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 12:47:03.554766 (+ 123us) client_negotiation.cc:240] Negotiation successful
0927 12:47:03.555061 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":106,"threads_started":1}
I20260927 12:47:03.557184 9837 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40691
I20260927 12:47:03.557390 9837 heartbeater.cc:464] Registering TS with master...
I20260927 12:47:03.557891 9837 heartbeater.cc:511] Master 127.0.98.62:40691 requested a full tablet report, sending...
I20260927 12:47:03.559291 9649 ts_manager.cc:194] Registered new tserver with Master: 02454500a8094200b291e5edee301b7b (127.0.98.1:43431)
I20260927 12:47:03.560429 9649 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:42895
W20260927 12:47:03.660889 9840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:47:03.661157 9840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:47:03.661211 9840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:47:03.665252 9840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:47:03.665313 9840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:47:03.665388 9840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:47:03.670125 9840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790513161160093-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790513161160093-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790513161160093-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:47:03.671262 9840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:47:03.672619 9840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:47:03.682312 9845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:47:03.682323 9848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:47:03.682379 9840 server_base.cc:1034] running on GCE node
W20260927 12:47:03.682328 9846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:47:03.682966 9840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:47:03.683634 9840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:47:03.684796 9840 hybrid_clock.cc:648] HybridClock initialized: now 1790513223684769 us; error 43 us; skew 500 ppm
I20260927 12:47:03.686666 9840 webserver.cc:460] Webserver started at http://127.0.98.2:45343/ using document root <none> and password file <none>
I20260927 12:47:03.687223 9840 fs_manager.cc:359] Metadata directory not provided
I20260927 12:47:03.687299 9840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:47:03.687487 9840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:47:03.689536 9840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790513161160093-392-0/minicluster-data/ts-1/data/instance:
uuid: "c6ccb2cf763c435faaadc3f7b9e23e11"
format_stamp: "Formatted at 2026-09-27 12:47:03 on dist-test-slave-lkn4"
server_key: "2780f7a6ccd3965c6c08bd650359e640"
server_key_iv: "7eade39f2cdec39dd6eecf07d85b6314"
server_key_version: "encryptionkey@0"
I20260927 12:47:03.690104 9840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790513161160093-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c6ccb2cf763c435faaadc3f7b9e23e11"
format_stamp: "Formatted at 2026-09-27 12:47:03 on dist-test-slave-lkn4"
server_key: "2780f7a6ccd3965c6c08bd650359e640"
server_key_iv: "7eade39f2cdec39dd6eecf07d85b6314"
server_key_version: "encryptionkey@0"
I20260927 12:47:03.693936 9840 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 12:47:03.696545 9854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:47:03.697929 9840 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:47:03.698071 9840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790513161160093-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790513161160093-392-0/minicluster-data/ts-1/wal
uuid: "c6ccb2cf763c435faaadc3f7b9e23e11"
format_stamp: "Formatted at 2026-09-27 12:47:03 on dist-test-slave-lkn4"
server_key: "2780f7a6ccd3965c6c08bd650359e640"
server_key_iv: "7eade39f2cdec39dd6eecf07d85b6314"
server_key_version: "encryptionkey@0"
I20260927 12:47:03.698187 9840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790513161160093-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790513161160093-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790513161160093-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:47:03.720788 9840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:47:03.721541 9840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:47:03.721722 9840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:47:03.722405 9840 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:47:03.723373 9840 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:47:03.723421 9840 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:47:03.723475 9840 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:47:03.723505 9840 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:47:03.732206 9840 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40431
I20260927 12:47:03.732270 9967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40431 every 8 connection(s)
I20260927 12:47:03.733158 9840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:47:03.740172 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 9840
I20260927 12:47:03.740291 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790513161160093-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:47:03.740571 392 external_mini_cluster.cc:1428] Setting key 0daadd8ce6f9bc764622974f2973cc6a
I20260927 12:47:03.742585 9688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:47:03.735067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43819 (local address 127.0.98.62:40691)
0927 12:47:03.735201 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 12:47:03.735205 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:47:03.736337 (+ 1132us) server_negotiation.cc:407] Connection header received
0927 12:47:03.737780 (+ 1443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:47:03.737784 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:47:03.737846 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:47:03.737926 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:47:03.738923 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:47:03.739442 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:47:03.740651 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:47:03.740911 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:47:03.741003 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:47:03.742156 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:47:03.742159 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:47:03.742161 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:47:03.742172 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:47:03.742194 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:47:03.742202 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:47:03.742258 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 12:47:03.742423 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 12:47:03.742475 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 12:47:03.742755 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790513161160093-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790513161160093-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790513161160093-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40691
--builtin_ntp_servers=127.0.98.20:35397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 12:47:03.743785 9970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:47:03.735468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40691 (local address 127.0.98.2:43819)
0927 12:47:03.736153 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0927 12:47:03.736198 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 12:47:03.737542 (+ 1344us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:47:03.738146 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:47:03.738154 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:47:03.738254 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:47:03.738773 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:47:03.738787 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:47:03.739565 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:47:03.739568 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:47:03.740530 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:47:03.740541 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:47:03.741063 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:47:03.741068 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:47:03.741230 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:47:03.741855 (+ 625us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 12:47:03.741856 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:47:03.741901 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 12:47:03.741976 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:47:03.742280 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:47:03.742290 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:47:03.742296 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 12:47:03.742432 (+ 136us) client_negotiation.cc:240] Negotiation successful
0927 12:47:03.742753 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":116,"threads_started":1}
I20260927 12:47:03.744972 9968 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40691
I20260927 12:47:03.745201 9968 heartbeater.cc:464] Registering TS with master...
I20260927 12:47:03.745705 9968 heartbeater.cc:511] Master 127.0.98.62:40691 requested a full tablet report, sending...
I20260927 12:47:03.746788 9649 ts_manager.cc:194] Registered new tserver with Master: c6ccb2cf763c435faaadc3f7b9e23e11 (127.0.98.2:40431)
I20260927 12:47:03.747341 9649 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:43819
W20260927 12:47:03.852459 9971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:47:03.852751 9971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:47:03.852803 9971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:47:03.856932 9971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:47:03.856997 9971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:47:03.857069 9971 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:47:03.861891 9971 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790513161160093-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790513161160093-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790513161160093-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:47:03.862972 9971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:47:03.864311 9971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:47:03.873924 9977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:47:03.873963 9976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:47:03.874245 9971 server_base.cc:1034] running on GCE node
W20260927 12:47:03.873930 9979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:47:03.874838 9971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:47:03.875478 9971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:47:03.876646 9971 hybrid_clock.cc:648] HybridClock initialized: now 1790513223876629 us; error 36 us; skew 500 ppm
I20260927 12:47:03.878717 9971 webserver.cc:460] Webserver started at http://127.0.98.3:33657/ using document root <none> and password file <none>
I20260927 12:47:03.879262 9971 fs_manager.cc:359] Metadata directory not provided
I20260927 12:47:03.879317 9971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:47:03.879478 9971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:47:03.881531 9971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790513161160093-392-0/minicluster-data/ts-2/data/instance:
uuid: "f1890452b53b43fabe4aa47825afffda"
format_stamp: "Formatted at 2026-09-27 12:47:03 on dist-test-slave-lkn4"
server_key: "ff2db28b18066bfc27efe200003d965a"
server_key_iv: "bd2ad7d3a8e0618a012fea90f737d825"
server_key_version: "encryptionkey@0"
I20260927 12:47:03.882117 9971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790513161160093-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f1890452b53b43fabe4aa47825afffda"
format_stamp: "Formatted at 2026-09-27 12:47:03 on dist-test-slave-lkn4"
server_key: "ff2db28b18066bfc27efe200003d965a"
server_key_iv: "bd2ad7d3a8e0618a012fea90f737d825"
server_key_version: "encryptionkey@0"
I20260927 12:47:03.885993 9971 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 12:47:03.888815 9985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:47:03.890286 9971 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:47:03.890395 9971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790513161160093-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790513161160093-392-0/minicluster-data/ts-2/wal
uuid: "f1890452b53b43fabe4aa47825afffda"
format_stamp: "Formatted at 2026-09-27 12:47:03 on dist-test-slave-lkn4"
server_key: "ff2db28b18066bfc27efe200003d965a"
server_key_iv: "bd2ad7d3a8e0618a012fea90f737d825"
server_key_version: "encryptionkey@0"
I20260927 12:47:03.890512 9971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790513161160093-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790513161160093-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790513161160093-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:47:03.902459 9971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:47:03.903095 9971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:47:03.903276 9971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:47:03.903834 9971 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:47:03.904750 9971 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:47:03.904796 9971 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:47:03.904834 9971 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:47:03.904848 9971 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:47:03.914235 9971 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35461
I20260927 12:47:03.914268 10098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35461 every 8 connection(s)
I20260927 12:47:03.915220 9971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:47:03.919054 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 9971
I20260927 12:47:03.919167 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790513161160093-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:47:03.919446 392 external_mini_cluster.cc:1428] Setting key d50798a1322c41d60dc5c82a2a17bc70
I20260927 12:47:03.925230 9688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:47:03.917329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50853 (local address 127.0.98.62:40691)
0927 12:47:03.917488 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 12:47:03.917493 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:47:03.918760 (+ 1267us) server_negotiation.cc:407] Connection header received
0927 12:47:03.920301 (+ 1541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:47:03.920304 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:47:03.920387 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:47:03.920469 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:47:03.921439 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:47:03.922061 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:47:03.923270 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:47:03.923492 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:47:03.923573 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:47:03.924630 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:47:03.924633 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:47:03.924635 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:47:03.924648 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:47:03.924669 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:47:03.924676 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:47:03.924751 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 12:47:03.925049 (+ 298us) server_negotiation.cc:299] Negotiation successful
0927 12:47:03.925105 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 12:47:03.926152 10101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:47:03.917824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40691 (local address 127.0.98.3:50853)
0927 12:47:03.918590 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0927 12:47:03.918642 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 12:47:03.920072 (+ 1430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:47:03.920631 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:47:03.920638 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:47:03.920732 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:47:03.921304 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:47:03.921317 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:47:03.922179 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:47:03.922181 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:47:03.923161 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:47:03.923170 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:47:03.923610 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:47:03.923613 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:47:03.923807 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:47:03.924367 (+ 560us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 12:47:03.924369 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:47:03.924410 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 12:47:03.924478 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:47:03.924847 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:47:03.924884 (+ 37us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:47:03.924890 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 12:47:03.925009 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 12:47:03.925275 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":171,"threads_started":1}
I20260927 12:47:03.927426 10099 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40691
I20260927 12:47:03.927680 10099 heartbeater.cc:464] Registering TS with master...
I20260927 12:47:03.928253 10099 heartbeater.cc:511] Master 127.0.98.62:40691 requested a full tablet report, sending...
I20260927 12:47:03.929283 9649 ts_manager.cc:194] Registered new tserver with Master: f1890452b53b43fabe4aa47825afffda (127.0.98.3:35461)
I20260927 12:47:03.929953 9649 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:50853
I20260927 12:47:03.933368 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 12:47:03.936136 9688 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:47:03.935234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48694 (local address 127.0.98.62:40691)
0927 12:47:03.935378 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 12:47:03.935383 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:47:03.935520 (+ 137us) server_negotiation.cc:407] Connection header received
0927 12:47:03.935626 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:47:03.935628 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:47:03.935735 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:47:03.935798 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:47:03.936012 (+ 214us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48694: BlockingRecv error: recv got EOF from 127.0.0.1:48694 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 12:47:03.936277 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 9709
I20260927 12:47:03.941179 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 9840
I20260927 12:47:03.945586 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 9971
I20260927 12:47:03.949951 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 9618
2026-09-27T12:47:03Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (840 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-27T12:47:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:47:03Z Disabled control of system clock
I20260927 12:47:03.977666 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790513161160093-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790513161160093-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790513161160093-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33211
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33211
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 12:47:04.090472 10110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:47:04.090746 10110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:47:04.090781 10110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:47:04.095927 10110 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:47:04.095999 10110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:47:04.096012 10110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:47:04.096029 10110 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:47:04.096043 10110 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:47:04.101089 10110 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790513161160093-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790513161160093-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33211
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10110
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790513161160093-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:47:04.102226 10110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:47:04.103616 10110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:47:04.111600 10118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:47:04.111649 10116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:47:04.111814 10110 server_base.cc:1034] running on GCE node
W20260927 12:47:04.111604 10115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:47:04.112556 10110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:47:04.113739 10110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:47:04.114907 10110 hybrid_clock.cc:648] HybridClock initialized: now 1790513224114874 us; error 46 us; skew 500 ppm
I20260927 12:47:04.116886 10110 webserver.cc:460] Webserver started at http://127.0.98.62:43855/ using document root <none> and password file <none>
I20260927 12:47:04.117473 10110 fs_manager.cc:359] Metadata directory not provided
I20260927 12:47:04.117538 10110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:47:04.117736 10110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:47:04.119920 10110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790513161160093-392-0/minicluster-data/master-0/data/instance:
uuid: "7afd6a533d454c5e9c1737ce033c1d41"
format_stamp: "Formatted at 2026-09-27 12:47:04 on dist-test-slave-lkn4"
server_key: "c68f5179cba67b84a6a2ad8a9b830912"
server_key_iv: "d8f02ab62a41384eb56dceb26659baaf"
server_key_version: "encryptionkey@0"
I20260927 12:47:04.120496 10110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790513161160093-392-0/minicluster-data/master-0/wal/instance:
uuid: "7afd6a533d454c5e9c1737ce033c1d41"
format_stamp: "Formatted at 2026-09-27 12:47:04 on dist-test-slave-lkn4"
server_key: "c68f5179cba67b84a6a2ad8a9b830912"
server_key_iv: "d8f02ab62a41384eb56dceb26659baaf"
server_key_version: "encryptionkey@0"
I20260927 12:47:04.124567 10110 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 12:47:04.127379 10124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:47:04.128934 10110 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:47:04.129081 10110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790513161160093-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790513161160093-392-0/minicluster-data/master-0/wal
uuid: "7afd6a533d454c5e9c1737ce033c1d41"
format_stamp: "Formatted at 2026-09-27 12:47:04 on dist-test-slave-lkn4"
server_key: "c68f5179cba67b84a6a2ad8a9b830912"
server_key_iv: "d8f02ab62a41384eb56dceb26659baaf"
server_key_version: "encryptionkey@0"
I20260927 12:47:04.129204 10110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790513161160093-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790513161160093-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790513161160093-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:47:04.133999 10110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:47:04.134693 10110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:47:04.134865 10110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:47:04.141387 10110 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33211
I20260927 12:47:04.141425 10176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33211 every 8 connection(s)
I20260927 12:47:04.142380 10110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:47:04.143432 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 10110
I20260927 12:47:04.143522 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790513161160093-392-0/minicluster-data/master-0/wal/instance
I20260927 12:47:04.143774 392 external_mini_cluster.cc:1428] Setting key eca57b53e18c51ae8c8887a0b1a92338
I20260927 12:47:04.145641 10177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:47:04.149719 10180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:47:04.145114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37704 (local address 127.0.98.62:33211)
0927 12:47:04.146306 (+ 1192us) server_negotiation.cc:206] Beginning negotiation
0927 12:47:04.146322 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 12:47:04.146359 (+ 37us) server_negotiation.cc:407] Connection header received
0927 12:47:04.147061 (+ 702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:47:04.147079 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:47:04.147425 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:47:04.147729 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:47:04.148519 (+ 790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:47:04.148524 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:47:04.148536 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:47:04.148546 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:47:04.148599 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:47:04.148615 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:47:04.148758 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 12:47:04.148826 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 12:47:04.149053 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":794,"thread_start_us":653,"threads_started":1}
I20260927 12:47:04.151574 10177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7afd6a533d454c5e9c1737ce033c1d41: Bootstrap starting.
I20260927 12:47:04.153775 10177 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7afd6a533d454c5e9c1737ce033c1d41: Neither blocks nor log segments found. Creating new log.
I20260927 12:47:04.154534 10177 log.cc:824] T 00000000000000000000000000000000 P 7afd6a533d454c5e9c1737ce033c1d41: Log is configured to *not* fsync() on all Append() calls
I20260927 12:47:04.156461 10177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7afd6a533d454c5e9c1737ce033c1d41: No bootstrap required, opened a new log
I20260927 12:47:04.160137 10177 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7afd6a533d454c5e9c1737ce033c1d41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7afd6a533d454c5e9c1737ce033c1d41" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33211 } }
I20260927 12:47:04.160457 10177 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7afd6a533d454c5e9c1737ce033c1d41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:47:04.160555 10177 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7afd6a533d454c5e9c1737ce033c1d41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7afd6a533d454c5e9c1737ce033c1d41, State: Initialized, Role: FOLLOWER
I20260927 12:47:04.161059 10177 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7afd6a533d454c5e9c1737ce033c1d41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7afd6a533d454c5e9c1737ce033c1d41" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33211 } }
I20260927 12:47:04.161204 10177 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7afd6a533d454c5e9c1737ce033c1d41 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:47:04.161248 10177 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7afd6a533d454c5e9c1737ce033c1d41 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:47:04.161325 10177 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7afd6a533d454c5e9c1737ce033c1d41 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:47:04.162254 10177 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7afd6a533d454c5e9c1737ce033c1d41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7afd6a533d454c5e9c1737ce033c1d41" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33211 } }
I20260927 12:47:04.162578 10177 leader_election.cc:302] T 00000000000000000000000000000000 P 7afd6a533d454c5e9c1737ce033c1d41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7afd6a533d454c5e9c1737ce033c1d41; no voters:
I20260927 12:47:04.162860 10177 leader_election.cc:288] T 00000000000000000000000000000000 P 7afd6a533d454c5e9c1737ce033c1d41 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:47:04.162966 10182 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7afd6a533d454c5e9c1737ce033c1d41 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:47:04.163184 10182 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7afd6a533d454c5e9c1737ce033c1d41 [term 1 LEADER]: Becoming Leader. State: Replica: 7afd6a533d454c5e9c1737ce033c1d41, State: Running, Role: LEADER
I20260927 12:47:04.163532 10182 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7afd6a533d454c5e9c1737ce033c1d41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7afd6a533d454c5e9c1737ce033c1d41" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33211 } }
I20260927 12:47:04.164078 10177 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7afd6a533d454c5e9c1737ce033c1d41 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:47:04.165160 10184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7afd6a533d454c5e9c1737ce033c1d41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7afd6a533d454c5e9c1737ce033c1d41. Latest consensus state: current_term: 1 leader_uuid: "7afd6a533d454c5e9c1737ce033c1d41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7afd6a533d454c5e9c1737ce033c1d41" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33211 } } }
I20260927 12:47:04.165187 10183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7afd6a533d454c5e9c1737ce033c1d41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7afd6a533d454c5e9c1737ce033c1d41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7afd6a533d454c5e9c1737ce033c1d41" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33211 } } }
I20260927 12:47:04.165293 10184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7afd6a533d454c5e9c1737ce033c1d41 [sys.catalog]: This master's current role is: LEADER
I20260927 12:47:04.165293 10183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7afd6a533d454c5e9c1737ce033c1d41 [sys.catalog]: This master's current role is: LEADER
I20260927 12:47:04.165725 10191 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:47:04.168619 10191 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:47:04.174997 10191 catalog_manager.cc:1347] Generated new cluster ID: eee537632604420bb45e9b5db2bde5b0
I20260927 12:47:04.175081 10191 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:47:04.184214 10191 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:47:04.184597 10191 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:47:04.195680 10191 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7afd6a533d454c5e9c1737ce033c1d41: Generated new TSK 0
I20260927 12:47:04.196452 10191 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:47:04.207068 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790513161160093-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790513161160093-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790513161160093-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33211
--builtin_ntp_servers=127.0.98.20:40093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 12:47:04.315618 10201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:47:04.315888 10201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:47:04.315935 10201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:47:04.320029 10201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:47:04.320099 10201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:47:04.320181 10201 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:47:04.325165 10201 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790513161160093-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790513161160093-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790513161160093-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:47:04.326326 10201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:47:04.327760 10201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:47:04.337527 10206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:47:04.337532 10207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:47:04.337572 10209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:47:04.337994 10201 server_base.cc:1034] running on GCE node
I20260927 12:47:04.338467 10201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:47:04.339088 10201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:47:04.340272 10201 hybrid_clock.cc:648] HybridClock initialized: now 1790513224340249 us; error 34 us; skew 500 ppm
I20260927 12:47:04.342010 10201 webserver.cc:460] Webserver started at http://127.0.98.1:38759/ using document root <none> and password file <none>
I20260927 12:47:04.342587 10201 fs_manager.cc:359] Metadata directory not provided
I20260927 12:47:04.342648 10201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:47:04.342821 10201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:47:04.344777 10201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790513161160093-392-0/minicluster-data/ts-0/data/instance:
uuid: "d9ced973fd9b4f7e8bd2a01af5b8cda1"
format_stamp: "Formatted at 2026-09-27 12:47:04 on dist-test-slave-lkn4"
server_key: "0b71ac98b4f263ad67b72e6d2cb7ce2e"
server_key_iv: "dbaee50cf8af5bb9efe60004cb7d68a5"
server_key_version: "encryptionkey@0"
I20260927 12:47:04.345286 10201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790513161160093-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d9ced973fd9b4f7e8bd2a01af5b8cda1"
format_stamp: "Formatted at 2026-09-27 12:47:04 on dist-test-slave-lkn4"
server_key: "0b71ac98b4f263ad67b72e6d2cb7ce2e"
server_key_iv: "dbaee50cf8af5bb9efe60004cb7d68a5"
server_key_version: "encryptionkey@0"
I20260927 12:47:04.348866 10201 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260927 12:47:04.351236 10215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:47:04.352468 10201 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 12:47:04.352571 10201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790513161160093-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790513161160093-392-0/minicluster-data/ts-0/wal
uuid: "d9ced973fd9b4f7e8bd2a01af5b8cda1"
format_stamp: "Formatted at 2026-09-27 12:47:04 on dist-test-slave-lkn4"
server_key: "0b71ac98b4f263ad67b72e6d2cb7ce2e"
server_key_iv: "dbaee50cf8af5bb9efe60004cb7d68a5"
server_key_version: "encryptionkey@0"
I20260927 12:47:04.352689 10201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790513161160093-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790513161160093-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790513161160093-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:47:04.357308 10201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:47:04.357890 10201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:47:04.358067 10201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:47:04.358609 10201 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:47:04.359494 10201 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:47:04.359539 10201 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:47:04.359580 10201 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:47:04.359608 10201 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:47:04.368059 10201 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46601
I20260927 12:47:04.368124 10328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46601 every 8 connection(s)
I20260927 12:47:04.369083 10201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:47:04.373032 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 10201
I20260927 12:47:04.373157 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790513161160093-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:47:04.373483 392 external_mini_cluster.cc:1428] Setting key 215b86b29ed849874d9d0447069de404
I20260927 12:47:04.375447 10180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:47:04.370895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45453 (local address 127.0.98.62:33211)
0927 12:47:04.371064 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 12:47:04.371068 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:47:04.372116 (+ 1048us) server_negotiation.cc:407] Connection header received
0927 12:47:04.373636 (+ 1520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:47:04.373639 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:47:04.373736 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:47:04.373815 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:47:04.375028 (+ 1213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:47:04.375031 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:47:04.375033 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:47:04.375043 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:47:04.375062 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:47:04.375070 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:47:04.375138 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 12:47:04.375312 (+ 174us) server_negotiation.cc:299] Negotiation successful
0927 12:47:04.375348 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 12:47:04.375618 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790513161160093-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790513161160093-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790513161160093-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33211
--builtin_ntp_servers=127.0.98.20:40093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 12:47:04.376593 10331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:47:04.371305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33211 (local address 127.0.98.1:45453)
0927 12:47:04.371965 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0927 12:47:04.372005 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 12:47:04.373411 (+ 1406us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:47:04.373992 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:47:04.374001 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:47:04.374144 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:47:04.374744 (+ 600us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 12:47:04.374746 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:47:04.374805 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 12:47:04.374893 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:47:04.375142 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:47:04.375150 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:47:04.375155 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 12:47:04.375352 (+ 197us) client_negotiation.cc:240] Negotiation successful
0927 12:47:04.375606 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":108,"threads_started":1}
I20260927 12:47:04.377869 10329 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33211
I20260927 12:47:04.378123 10329 heartbeater.cc:464] Registering TS with master...
I20260927 12:47:04.378547 10329 heartbeater.cc:511] Master 127.0.98.62:33211 requested a full tablet report, sending...
I20260927 12:47:04.380278 10141 ts_manager.cc:194] Registered new tserver with Master: d9ced973fd9b4f7e8bd2a01af5b8cda1 (127.0.98.1:46601)
W20260927 12:47:04.486631 10332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:47:04.486912 10332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:47:04.486953 10332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:47:04.490967 10332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:47:04.491024 10332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:47:04.491081 10332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:47:04.495942 10332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790513161160093-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790513161160093-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790513161160093-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:47:04.496948 10332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:47:04.498281 10332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 12:47:04.507593 10332 server_base.cc:1034] running on GCE node
W20260927 12:47:04.507555 10338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:47:04.507597 10340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:47:04.507550 10337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:47:04.508183 10332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:47:04.508903 10332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:47:04.510087 10332 hybrid_clock.cc:648] HybridClock initialized: now 1790513224510067 us; error 29 us; skew 500 ppm
I20260927 12:47:04.511914 10332 webserver.cc:460] Webserver started at http://127.0.98.2:43825/ using document root <none> and password file <none>
I20260927 12:47:04.512480 10332 fs_manager.cc:359] Metadata directory not provided
I20260927 12:47:04.512542 10332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:47:04.512717 10332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:47:04.514776 10332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790513161160093-392-0/minicluster-data/ts-1/data/instance:
uuid: "4d813e5c41a7403f8104f37a20fc9dcd"
format_stamp: "Formatted at 2026-09-27 12:47:04 on dist-test-slave-lkn4"
server_key: "c1b0e0bb632943d496a39e9304301439"
server_key_iv: "53ac6d6675f23f7dbd33ace50ae5356e"
server_key_version: "encryptionkey@0"
I20260927 12:47:04.515298 10332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790513161160093-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4d813e5c41a7403f8104f37a20fc9dcd"
format_stamp: "Formatted at 2026-09-27 12:47:04 on dist-test-slave-lkn4"
server_key: "c1b0e0bb632943d496a39e9304301439"
server_key_iv: "53ac6d6675f23f7dbd33ace50ae5356e"
server_key_version: "encryptionkey@0"
I20260927 12:47:04.518765 10332 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:47:04.521008 10346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:47:04.522434 10332 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:47:04.522557 10332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790513161160093-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790513161160093-392-0/minicluster-data/ts-1/wal
uuid: "4d813e5c41a7403f8104f37a20fc9dcd"
format_stamp: "Formatted at 2026-09-27 12:47:04 on dist-test-slave-lkn4"
server_key: "c1b0e0bb632943d496a39e9304301439"
server_key_iv: "53ac6d6675f23f7dbd33ace50ae5356e"
server_key_version: "encryptionkey@0"
I20260927 12:47:04.522653 10332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790513161160093-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790513161160093-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790513161160093-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:47:04.527297 10332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:47:04.527933 10332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:47:04.528091 10332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:47:04.528628 10332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:47:04.529561 10332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:47:04.529605 10332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:47:04.529657 10332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:47:04.529691 10332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:47:04.538487 10332 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41217
I20260927 12:47:04.538532 10459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41217 every 8 connection(s)
I20260927 12:47:04.539535 10332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:47:04.542204 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 10332
I20260927 12:47:04.542341 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790513161160093-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:47:04.542650 392 external_mini_cluster.cc:1428] Setting key eb9aca91490369febc89b4b92e1a3e13
I20260927 12:47:04.544740 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790513161160093-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790513161160093-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790513161160093-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33211
--builtin_ntp_servers=127.0.98.20:40093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 12:47:04.546211 10180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:47:04.541344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51919 (local address 127.0.98.62:33211)
0927 12:47:04.541464 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 12:47:04.541468 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:47:04.542835 (+ 1367us) server_negotiation.cc:407] Connection header received
0927 12:47:04.544421 (+ 1586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:47:04.544425 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:47:04.544467 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:47:04.544540 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:47:04.545762 (+ 1222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:47:04.545766 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:47:04.545769 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:47:04.545782 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:47:04.545803 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:47:04.545813 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:47:04.545885 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 12:47:04.546019 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 12:47:04.546084 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 12:47:04.547173 10462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:47:04.541728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33211 (local address 127.0.98.2:51919)
0927 12:47:04.542592 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0927 12:47:04.542712 (+ 120us) client_negotiation.cc:174] Beginning negotiation
0927 12:47:04.544181 (+ 1469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:47:04.544721 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:47:04.544729 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:47:04.544881 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:47:04.545497 (+ 616us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 12:47:04.545498 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:47:04.545543 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 12:47:04.545616 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:47:04.545880 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:47:04.545887 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:47:04.545893 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 12:47:04.546077 (+ 184us) client_negotiation.cc:240] Negotiation successful
0927 12:47:04.546394 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":142,"threads_started":1}
I20260927 12:47:04.548305 10460 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33211
I20260927 12:47:04.548516 10460 heartbeater.cc:464] Registering TS with master...
I20260927 12:47:04.548946 10460 heartbeater.cc:511] Master 127.0.98.62:33211 requested a full tablet report, sending...
I20260927 12:47:04.550113 10141 ts_manager.cc:194] Registered new tserver with Master: 4d813e5c41a7403f8104f37a20fc9dcd (127.0.98.2:41217)
W20260927 12:47:04.654100 10463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:47:04.654402 10463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:47:04.654444 10463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:47:04.658757 10463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:47:04.658830 10463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:47:04.658895 10463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:47:04.663951 10463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790513161160093-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790513161160093-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33211
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790513161160093-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:47:04.665155 10463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:47:04.666746 10463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:47:04.676509 10468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:47:04.676509 10469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:47:04.676512 10471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:47:04.676860 10463 server_base.cc:1034] running on GCE node
I20260927 12:47:04.677260 10463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:47:04.677932 10463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:47:04.679109 10463 hybrid_clock.cc:648] HybridClock initialized: now 1790513224679084 us; error 41 us; skew 500 ppm
I20260927 12:47:04.681285 10463 webserver.cc:460] Webserver started at http://127.0.98.3:44385/ using document root <none> and password file <none>
I20260927 12:47:04.681877 10463 fs_manager.cc:359] Metadata directory not provided
I20260927 12:47:04.681946 10463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:47:04.682188 10463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:47:04.684370 10463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790513161160093-392-0/minicluster-data/ts-2/data/instance:
uuid: "aa62fdd693dd4286a60a2606f271a366"
format_stamp: "Formatted at 2026-09-27 12:47:04 on dist-test-slave-lkn4"
server_key: "df0ac97fe54d38012aa1a8146226bd35"
server_key_iv: "b979c375faba8a8b28677cdb89e08334"
server_key_version: "encryptionkey@0"
I20260927 12:47:04.684988 10463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790513161160093-392-0/minicluster-data/ts-2/wal/instance:
uuid: "aa62fdd693dd4286a60a2606f271a366"
format_stamp: "Formatted at 2026-09-27 12:47:04 on dist-test-slave-lkn4"
server_key: "df0ac97fe54d38012aa1a8146226bd35"
server_key_iv: "b979c375faba8a8b28677cdb89e08334"
server_key_version: "encryptionkey@0"
I20260927 12:47:04.689318 10463 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 12:47:04.692951 10477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:47:04.694624 10463 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:47:04.694793 10463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790513161160093-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790513161160093-392-0/minicluster-data/ts-2/wal
uuid: "aa62fdd693dd4286a60a2606f271a366"
format_stamp: "Formatted at 2026-09-27 12:47:04 on dist-test-slave-lkn4"
server_key: "df0ac97fe54d38012aa1a8146226bd35"
server_key_iv: "b979c375faba8a8b28677cdb89e08334"
server_key_version: "encryptionkey@0"
I20260927 12:47:04.694940 10463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790513161160093-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790513161160093-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790513161160093-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:47:04.699541 10463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:47:04.700177 10463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:47:04.700343 10463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:47:04.700910 10463 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:47:04.701788 10463 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:47:04.701831 10463 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:47:04.701874 10463 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:47:04.701900 10463 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:47:04.709939 10463 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46171
I20260927 12:47:04.709993 10590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46171 every 8 connection(s)
I20260927 12:47:04.710871 10463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:47:04.717108 10180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:47:04.712722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55217 (local address 127.0.98.62:33211)
0927 12:47:04.712847 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 12:47:04.712852 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:47:04.713930 (+ 1078us) server_negotiation.cc:407] Connection header received
0927 12:47:04.715335 (+ 1405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:47:04.715338 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:47:04.715379 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:47:04.715455 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:47:04.716605 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:47:04.716609 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:47:04.716611 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:47:04.716626 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:47:04.716660 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:47:04.716678 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:47:04.716781 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 12:47:04.716950 (+ 169us) server_negotiation.cc:299] Negotiation successful
0927 12:47:04.716995 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 12:47:04.717998 10593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:47:04.713053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33211 (local address 127.0.98.3:55217)
0927 12:47:04.713779 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0927 12:47:04.713821 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 12:47:04.715118 (+ 1297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:47:04.715611 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:47:04.715618 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:47:04.715774 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:47:04.716314 (+ 540us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 12:47:04.716315 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:47:04.716356 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 12:47:04.716424 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:47:04.716785 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:47:04.716792 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:47:04.716797 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 12:47:04.716942 (+ 145us) client_negotiation.cc:240] Negotiation successful
0927 12:47:04.717153 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":122,"threads_started":1}
I20260927 12:47:04.719161 10591 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33211
I20260927 12:47:04.719396 10591 heartbeater.cc:464] Registering TS with master...
I20260927 12:47:04.719864 10591 heartbeater.cc:511] Master 127.0.98.62:33211 requested a full tablet report, sending...
I20260927 12:47:04.721014 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 10463
I20260927 12:47:04.721052 10141 ts_manager.cc:194] Registered new tserver with Master: aa62fdd693dd4286a60a2606f271a366 (127.0.98.3:46171)
I20260927 12:47:04.721109 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790513161160093-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:47:04.721415 392 external_mini_cluster.cc:1428] Setting key f520e355cf67122b008b823e480c971f
I20260927 12:47:04.723412 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 12:47:04.726394 10180 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:47:04.725354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37706 (local address 127.0.98.62:33211)
0927 12:47:04.725486 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 12:47:04.725491 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:47:04.725756 (+ 265us) server_negotiation.cc:407] Connection header received
0927 12:47:04.725900 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:47:04.725903 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:47:04.725944 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:47:04.726016 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:47:04.726283 (+ 267us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37706: BlockingRecv error: recv got EOF from 127.0.0.1:37706 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 12:47:04.726557 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 10201
I20260927 12:47:04.731333 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 10332
I20260927 12:47:04.735895 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 10463
I20260927 12:47:04.740226 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 10110
2026-09-27T12:47:04Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (790 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:47:04 dist-test-slave-lkn4 krb5kdc[10601](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:47:04 dist-test-slave-lkn4 krb5kdc[10601](info): set up 2 sockets
Sep 27 12:47:04 dist-test-slave-lkn4 krb5kdc[10601](info): commencing operation
krb5kdc: starting...
W20260927 12:47:05.702797 392 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/krb5kdc/test-user.keytab.
Sep 27 12:47:05 dist-test-slave-lkn4 krb5kdc[10601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:47:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:47:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:47:05.829558 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33313
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33313
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:47:05.939538 10614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:47:05.939821 10614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:47:05.939860 10614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:47:05.943925 10614 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:47:05.943989 10614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:47:05.944008 10614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:47:05.944023 10614 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:47:05.944038 10614 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:47:05.949394 10614 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:47:05.950628 10614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:47:05.951661 10614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:47:05.960212 10619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:47:05.960242 10622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:47:05.960227 10620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:47:05.960278 10614 server_base.cc:1034] running on GCE node
I20260927 12:47:05.961100 10614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:47:05.962263 10614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:47:05.963430 10614 hybrid_clock.cc:648] HybridClock initialized: now 1790513225963411 us; error 36 us; skew 500 ppm
Sep 27 12:47:05 dist-test-slave-lkn4 krb5kdc[10601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:47:05.966353 10614 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:47:05.967278 10614 webserver.cc:460] Webserver started at http://127.0.98.62:38219/ using document root <none> and password file <none>
I20260927 12:47:05.967821 10614 fs_manager.cc:359] Metadata directory not provided
I20260927 12:47:05.967888 10614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:47:05.968084 10614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:47:05.970249 10614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/minicluster-data/master-0/data/instance:
uuid: "11c569487cee436097c31d4f2cfe4c66"
format_stamp: "Formatted at 2026-09-27 12:47:05 on dist-test-slave-lkn4"
server_key: "967bd5ea45eb9fb6588da02c6e48c6c0"
server_key_iv: "79b6cdf301b77e4c928becb1f8223687"
server_key_version: "encryptionkey@0"
I20260927 12:47:05.970840 10614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/minicluster-data/master-0/wal/instance:
uuid: "11c569487cee436097c31d4f2cfe4c66"
format_stamp: "Formatted at 2026-09-27 12:47:05 on dist-test-slave-lkn4"
server_key: "967bd5ea45eb9fb6588da02c6e48c6c0"
server_key_iv: "79b6cdf301b77e4c928becb1f8223687"
server_key_version: "encryptionkey@0"
I20260927 12:47:05.974821 10614 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 12:47:05.977485 10629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:47:05.978915 10614 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:47:05.979044 10614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/minicluster-data/master-0/wal
uuid: "11c569487cee436097c31d4f2cfe4c66"
format_stamp: "Formatted at 2026-09-27 12:47:05 on dist-test-slave-lkn4"
server_key: "967bd5ea45eb9fb6588da02c6e48c6c0"
server_key_iv: "79b6cdf301b77e4c928becb1f8223687"
server_key_version: "encryptionkey@0"
I20260927 12:47:05.979151 10614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:47:05.999744 10614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:47:06.003015 10614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:47:06.003201 10614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:47:06.009944 10614 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33313
I20260927 12:47:06.010226 10681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33313 every 8 connection(s)
I20260927 12:47:06.010928 10614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:47:06.013697 10682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:47:06.015925 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 10614
I20260927 12:47:06.016037 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/minicluster-data/master-0/wal/instance
I20260927 12:47:06.016319 392 external_mini_cluster.cc:1428] Setting key bc51ffc06fc1b59c72a78a064462ecea
I20260927 12:47:06.019999 10682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11c569487cee436097c31d4f2cfe4c66: Bootstrap starting.
I20260927 12:47:06.022574 10682 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11c569487cee436097c31d4f2cfe4c66: Neither blocks nor log segments found. Creating new log.
I20260927 12:47:06.023351 10682 log.cc:824] T 00000000000000000000000000000000 P 11c569487cee436097c31d4f2cfe4c66: Log is configured to *not* fsync() on all Append() calls
I20260927 12:47:06.025511 10682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11c569487cee436097c31d4f2cfe4c66: No bootstrap required, opened a new log
Sep 27 12:47:06 dist-test-slave-lkn4 krb5kdc[10601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:47:06.029985 10682 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11c569487cee436097c31d4f2cfe4c66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11c569487cee436097c31d4f2cfe4c66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33313 } }
I20260927 12:47:06.030328 10682 raft_consensus.cc:374] T 00000000000000000000000000000000 P 11c569487cee436097c31d4f2cfe4c66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:47:06.030393 10682 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11c569487cee436097c31d4f2cfe4c66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11c569487cee436097c31d4f2cfe4c66, State: Initialized, Role: FOLLOWER
I20260927 12:47:06.030911 10682 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11c569487cee436097c31d4f2cfe4c66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11c569487cee436097c31d4f2cfe4c66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33313 } }
I20260927 12:47:06.031073 10682 raft_consensus.cc:388] T 00000000000000000000000000000000 P 11c569487cee436097c31d4f2cfe4c66 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:47:06.031136 10682 raft_consensus.cc:482] T 00000000000000000000000000000000 P 11c569487cee436097c31d4f2cfe4c66 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:47:06.031231 10682 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11c569487cee436097c31d4f2cfe4c66 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:47:06.031668 10685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:47:06.017814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33518 (local address 127.0.98.62:33313)
0927 12:47:06.018234 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0927 12:47:06.018252 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 12:47:06.018304 (+ 52us) server_negotiation.cc:407] Connection header received
0927 12:47:06.019082 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:47:06.019106 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:47:06.019544 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:47:06.019911 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:47:06.020700 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:47:06.022182 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:47:06.023128 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:47:06.023812 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:47:06.023928 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:47:06.026331 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:47:06.026337 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:47:06.026353 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:47:06.026382 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:47:06.028384 (+ 2002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:47:06.028807 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:47:06.028813 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:47:06.028820 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:47:06.028964 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:47:06.029293 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:47:06.029297 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:47:06.029298 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:47:06.029815 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:47:06.030021 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0927 12:47:06.030325 (+ 304us) server_negotiation.cc:299] Negotiation successful
0927 12:47:06.030640 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":118,"threads_started":1}
I20260927 12:47:06.032457 10682 raft_consensus.cc:504] T 00000000000000000000000000000000 P 11c569487cee436097c31d4f2cfe4c66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11c569487cee436097c31d4f2cfe4c66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33313 } }
I20260927 12:47:06.032832 10682 leader_election.cc:302] T 00000000000000000000000000000000 P 11c569487cee436097c31d4f2cfe4c66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 11c569487cee436097c31d4f2cfe4c66; no voters:
I20260927 12:47:06.033165 10682 leader_election.cc:288] T 00000000000000000000000000000000 P 11c569487cee436097c31d4f2cfe4c66 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:47:06.033293 10687 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 11c569487cee436097c31d4f2cfe4c66 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:47:06.033492 10687 raft_consensus.cc:686] T 00000000000000000000000000000000 P 11c569487cee436097c31d4f2cfe4c66 [term 1 LEADER]: Becoming Leader. State: Replica: 11c569487cee436097c31d4f2cfe4c66, State: Running, Role: LEADER
I20260927 12:47:06.033806 10687 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11c569487cee436097c31d4f2cfe4c66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11c569487cee436097c31d4f2cfe4c66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33313 } }
I20260927 12:47:06.034503 10682 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11c569487cee436097c31d4f2cfe4c66 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:47:06.034981 10687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11c569487cee436097c31d4f2cfe4c66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11c569487cee436097c31d4f2cfe4c66. Latest consensus state: current_term: 1 leader_uuid: "11c569487cee436097c31d4f2cfe4c66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11c569487cee436097c31d4f2cfe4c66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33313 } } }
I20260927 12:47:06.035094 10687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11c569487cee436097c31d4f2cfe4c66 [sys.catalog]: This master's current role is: LEADER
I20260927 12:47:06.035539 10689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11c569487cee436097c31d4f2cfe4c66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "11c569487cee436097c31d4f2cfe4c66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11c569487cee436097c31d4f2cfe4c66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33313 } } }
I20260927 12:47:06.035694 10689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11c569487cee436097c31d4f2cfe4c66 [sys.catalog]: This master's current role is: LEADER
I20260927 12:47:06.035987 10692 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:47:06.039324 10692 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:47:06.044610 10692 catalog_manager.cc:1347] Generated new cluster ID: c118c26b8a7c4c1cbc2e425e38ab1222
I20260927 12:47:06.044689 10692 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:47:06.060164 10692 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:47:06.061201 10692 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:47:06.076874 10692 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 11c569487cee436097c31d4f2cfe4c66: Generated new TSK 0
I20260927 12:47:06.077656 10692 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 12:47:06.108816 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33313
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:47:06.249636 10708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:47:06.249974 10708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:47:06.250032 10708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:47:06.254452 10708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:47:06.254556 10708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:47:06.254638 10708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:47:06.260421 10708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:47:06.261516 10708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:47:06.262588 10708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:47:06.272105 10713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:47:06.272181 10716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:47:06.272145 10714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:47:06.272244 10708 server_base.cc:1034] running on GCE node
I20260927 12:47:06.272743 10708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:47:06.273353 10708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:47:06.274516 10708 hybrid_clock.cc:648] HybridClock initialized: now 1790513226274496 us; error 30 us; skew 500 ppm
Sep 27 12:47:06 dist-test-slave-lkn4 krb5kdc[10601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513226, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:47:06.282317 10708 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 12:47:06.283715 10708 webserver.cc:460] Webserver started at http://127.0.98.1:39127/ using document root <none> and password file <none>
I20260927 12:47:06.284314 10708 fs_manager.cc:359] Metadata directory not provided
I20260927 12:47:06.284402 10708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:47:06.284652 10708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:47:06.286968 10708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/minicluster-data/ts-0/data/instance:
uuid: "723197480f2e4ac58676b8d5ecacdd4d"
format_stamp: "Formatted at 2026-09-27 12:47:06 on dist-test-slave-lkn4"
server_key: "e2510be69cfea613bc46564f9846fb78"
server_key_iv: "eda95090c2f79aa1975baffb6973c469"
server_key_version: "encryptionkey@0"
I20260927 12:47:06.287583 10708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/minicluster-data/ts-0/wal/instance:
uuid: "723197480f2e4ac58676b8d5ecacdd4d"
format_stamp: "Formatted at 2026-09-27 12:47:06 on dist-test-slave-lkn4"
server_key: "e2510be69cfea613bc46564f9846fb78"
server_key_iv: "eda95090c2f79aa1975baffb6973c469"
server_key_version: "encryptionkey@0"
I20260927 12:47:06.291755 10708 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 12:47:06.294450 10723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:47:06.295835 10708 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:47:06.295966 10708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/minicluster-data/ts-0/wal
uuid: "723197480f2e4ac58676b8d5ecacdd4d"
format_stamp: "Formatted at 2026-09-27 12:47:06 on dist-test-slave-lkn4"
server_key: "e2510be69cfea613bc46564f9846fb78"
server_key_iv: "eda95090c2f79aa1975baffb6973c469"
server_key_version: "encryptionkey@0"
I20260927 12:47:06.296084 10708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:47:06.309895 10708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:47:06.313405 10708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:47:06.313580 10708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:47:06.314255 10708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:47:06.315071 10708 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:47:06.315119 10708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:47:06.315171 10708 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:47:06.315196 10708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:47:06.323477 10708 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46315
I20260927 12:47:06.323531 10836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46315 every 8 connection(s)
I20260927 12:47:06.324385 10708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:47:06.326470 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 10708
I20260927 12:47:06.326567 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:47:06.326853 392 external_mini_cluster.cc:1428] Setting key c87b21ccb6d48c39966c7c65b26cd152
Sep 27 12:47:06 dist-test-slave-lkn4 krb5kdc[10601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513226, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 12:47:06.338452 10685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:47:06.326228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37785 (local address 127.0.98.62:33313)
0927 12:47:06.326397 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 12:47:06.326402 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:47:06.327430 (+ 1028us) server_negotiation.cc:407] Connection header received
0927 12:47:06.328906 (+ 1476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:47:06.328910 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:47:06.328964 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:47:06.329040 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:47:06.330308 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:47:06.330854 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:47:06.331997 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:47:06.332222 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:47:06.332311 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:47:06.334964 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:47:06.334967 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:47:06.334969 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:47:06.334995 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:47:06.336384 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:47:06.336911 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:47:06.336914 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:47:06.336915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:47:06.336990 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:47:06.337306 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:47:06.337310 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:47:06.337311 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:47:06.337496 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:47:06.337580 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 12:47:06.338140 (+ 560us) server_negotiation.cc:299] Negotiation successful
0927 12:47:06.338280 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 12:47:06.339330 10839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:47:06.326659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33313 (local address 127.0.98.1:37785)
0927 12:47:06.327255 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0927 12:47:06.327311 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 12:47:06.328693 (+ 1382us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:47:06.329243 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:47:06.329251 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:47:06.329614 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:47:06.330166 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:47:06.330183 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:47:06.330974 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:47:06.330978 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:47:06.331881 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:47:06.331890 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:47:06.332364 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:47:06.332368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:47:06.332557 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:47:06.333192 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:47:06.333208 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:47:06.334804 (+ 1596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:47:06.336487 (+ 1683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:47:06.336494 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:47:06.336511 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:47:06.336810 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:47:06.337069 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:47:06.337072 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:47:06.337075 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:47:06.337215 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:47:06.337591 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:47:06.337598 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:47:06.337868 (+ 270us) client_negotiation.cc:769] Sending connection context
0927 12:47:06.338033 (+ 165us) client_negotiation.cc:240] Negotiation successful
0927 12:47:06.338409 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":105,"threads_started":1}
I20260927 12:47:06.340691 10837 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33313
I20260927 12:47:06.340997 10837 heartbeater.cc:464] Registering TS with master...
I20260927 12:47:06.341653 10837 heartbeater.cc:511] Master 127.0.98.62:33313 requested a full tablet report, sending...
I20260927 12:47:06.343400 10646 ts_manager.cc:194] Registered new tserver with Master: 723197480f2e4ac58676b8d5ecacdd4d (127.0.98.1:46315)
I20260927 12:47:06.344556 10646 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37785
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 12:47:06.357105 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33313
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:47:06.467449 10842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:47:06.467717 10842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:47:06.467759 10842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:47:06.471742 10842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:47:06.471798 10842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:47:06.471856 10842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:47:06.477124 10842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:47:06.478197 10842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:47:06.479177 10842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:47:06.488818 10850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:47:06.488919 10842 server_base.cc:1034] running on GCE node
W20260927 12:47:06.488729 10847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:47:06.488726 10848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:47:06.489471 10842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:47:06.490141 10842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:47:06.491320 10842 hybrid_clock.cc:648] HybridClock initialized: now 1790513226491297 us; error 35 us; skew 500 ppm
Sep 27 12:47:06 dist-test-slave-lkn4 krb5kdc[10601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513226, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:47:06.494220 10842 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 12:47:06.495110 10842 webserver.cc:460] Webserver started at http://127.0.98.2:45123/ using document root <none> and password file <none>
I20260927 12:47:06.495649 10842 fs_manager.cc:359] Metadata directory not provided
I20260927 12:47:06.495723 10842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:47:06.495932 10842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:47:06.497969 10842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/minicluster-data/ts-1/data/instance:
uuid: "fe4599b667cc4d299709d9f8acd0bccc"
format_stamp: "Formatted at 2026-09-27 12:47:06 on dist-test-slave-lkn4"
server_key: "c7c63f29ed0445e96a776b0fb6471d45"
server_key_iv: "00ec212139530a16bfa13c2a2cd85ea3"
server_key_version: "encryptionkey@0"
I20260927 12:47:06.498533 10842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/minicluster-data/ts-1/wal/instance:
uuid: "fe4599b667cc4d299709d9f8acd0bccc"
format_stamp: "Formatted at 2026-09-27 12:47:06 on dist-test-slave-lkn4"
server_key: "c7c63f29ed0445e96a776b0fb6471d45"
server_key_iv: "00ec212139530a16bfa13c2a2cd85ea3"
server_key_version: "encryptionkey@0"
I20260927 12:47:06.502363 10842 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 12:47:06.505148 10857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:47:06.506678 10842 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:47:06.506846 10842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/minicluster-data/ts-1/wal
uuid: "fe4599b667cc4d299709d9f8acd0bccc"
format_stamp: "Formatted at 2026-09-27 12:47:06 on dist-test-slave-lkn4"
server_key: "c7c63f29ed0445e96a776b0fb6471d45"
server_key_iv: "00ec212139530a16bfa13c2a2cd85ea3"
server_key_version: "encryptionkey@0"
I20260927 12:47:06.506955 10842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:47:06.528805 10842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:47:06.532198 10842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:47:06.532392 10842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:47:06.532987 10842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:47:06.533862 10842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:47:06.533910 10842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:47:06.533967 10842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:47:06.533996 10842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:47:06.542694 10842 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36153
I20260927 12:47:06.542716 10970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36153 every 8 connection(s)
I20260927 12:47:06.543599 10842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 12:47:06 dist-test-slave-lkn4 krb5kdc[10601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513226, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:47:06.553874 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 10842
I20260927 12:47:06.553993 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:47:06.554339 392 external_mini_cluster.cc:1428] Setting key edec1503c72e6fc3405d41259c6d376f
I20260927 12:47:06.557734 10685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:47:06.545312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53551 (local address 127.0.98.62:33313)
0927 12:47:06.545468 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 12:47:06.545472 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:47:06.546403 (+ 931us) server_negotiation.cc:407] Connection header received
0927 12:47:06.548014 (+ 1611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:47:06.548018 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:47:06.548072 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:47:06.548145 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:47:06.549313 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:47:06.549837 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:47:06.550989 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:47:06.551215 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:47:06.551302 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:47:06.553689 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:47:06.553692 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:47:06.553694 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:47:06.553720 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:47:06.555292 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:47:06.555877 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:47:06.555880 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:47:06.555881 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:47:06.555958 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:47:06.556369 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:47:06.556372 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:47:06.556374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:47:06.556582 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:47:06.556679 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 12:47:06.557399 (+ 720us) server_negotiation.cc:299] Negotiation successful
0927 12:47:06.557552 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 12:47:06.558593 10973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:47:06.545761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33313 (local address 127.0.98.2:53551)
0927 12:47:06.546255 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0927 12:47:06.546300 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 12:47:06.547799 (+ 1499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:47:06.548325 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:47:06.548332 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:47:06.548693 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:47:06.549170 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:47:06.549184 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:47:06.549949 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:47:06.549951 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:47:06.550878 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:47:06.550888 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:47:06.551358 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:47:06.551362 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:47:06.551516 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:47:06.552073 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:47:06.552095 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:47:06.553523 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:47:06.555414 (+ 1891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:47:06.555419 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:47:06.555432 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:47:06.555781 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:47:06.556044 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:47:06.556046 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:47:06.556048 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:47:06.556256 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:47:06.556713 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:47:06.556720 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:47:06.557048 (+ 328us) client_negotiation.cc:769] Sending connection context
0927 12:47:06.557294 (+ 246us) client_negotiation.cc:240] Negotiation successful
0927 12:47:06.557655 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":96,"threads_started":1}
I20260927 12:47:06.559769 10971 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33313
I20260927 12:47:06.560002 10971 heartbeater.cc:464] Registering TS with master...
I20260927 12:47:06.560523 10971 heartbeater.cc:511] Master 127.0.98.62:33313 requested a full tablet report, sending...
I20260927 12:47:06.561537 10646 ts_manager.cc:194] Registered new tserver with Master: fe4599b667cc4d299709d9f8acd0bccc (127.0.98.2:36153)
I20260927 12:47:06.562323 10646 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53551
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 12:47:06.581498 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33313
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:47:06.693715 10976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:47:06.694005 10976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:47:06.694083 10976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:47:06.698292 10976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:47:06.698359 10976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:47:06.698421 10976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:47:06.704053 10976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:47:06.705137 10976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:47:06.706164 10976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:47:06.715713 10982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:47:06.715838 10984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:47:06.715859 10981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:47:06.715993 10976 server_base.cc:1034] running on GCE node
I20260927 12:47:06.716377 10976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:47:06.716992 10976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:47:06.718110 10976 hybrid_clock.cc:648] HybridClock initialized: now 1790513226718074 us; error 49 us; skew 500 ppm
Sep 27 12:47:06 dist-test-slave-lkn4 krb5kdc[10601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513226, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:47:06.720902 10976 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 12:47:06.721767 10976 webserver.cc:460] Webserver started at http://127.0.98.3:39619/ using document root <none> and password file <none>
I20260927 12:47:06.722357 10976 fs_manager.cc:359] Metadata directory not provided
I20260927 12:47:06.722419 10976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:47:06.722604 10976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:47:06.724658 10976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/minicluster-data/ts-2/data/instance:
uuid: "006df6a8254c4409ac29acf54369e1d1"
format_stamp: "Formatted at 2026-09-27 12:47:06 on dist-test-slave-lkn4"
server_key: "283c9c5de9f8c1eee1c4d24fd9df90df"
server_key_iv: "bbf0338e71fa13c9d444f13ccd74d5ab"
server_key_version: "encryptionkey@0"
I20260927 12:47:06.725153 10976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/minicluster-data/ts-2/wal/instance:
uuid: "006df6a8254c4409ac29acf54369e1d1"
format_stamp: "Formatted at 2026-09-27 12:47:06 on dist-test-slave-lkn4"
server_key: "283c9c5de9f8c1eee1c4d24fd9df90df"
server_key_iv: "bbf0338e71fa13c9d444f13ccd74d5ab"
server_key_version: "encryptionkey@0"
I20260927 12:47:06.728945 10976 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 12:47:06.731701 10991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:47:06.733062 10976 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 12:47:06.733176 10976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/minicluster-data/ts-2/wal
uuid: "006df6a8254c4409ac29acf54369e1d1"
format_stamp: "Formatted at 2026-09-27 12:47:06 on dist-test-slave-lkn4"
server_key: "283c9c5de9f8c1eee1c4d24fd9df90df"
server_key_iv: "bbf0338e71fa13c9d444f13ccd74d5ab"
server_key_version: "encryptionkey@0"
I20260927 12:47:06.733275 10976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:47:06.752123 10976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:47:06.755267 10976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:47:06.755421 10976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:47:06.755981 10976 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:47:06.756858 10976 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:47:06.756904 10976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:47:06.756958 10976 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:47:06.756986 10976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:47:06.765470 10976 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43049
I20260927 12:47:06.765533 11104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43049 every 8 connection(s)
I20260927 12:47:06.766398 10976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:47:06.767899 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 10976
I20260927 12:47:06.767998 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:47:06.768317 392 external_mini_cluster.cc:1428] Setting key 0216b677c3d2ebc4cbeef865f3f5baf5
Sep 27 12:47:06 dist-test-slave-lkn4 krb5kdc[10601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513226, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:47:06.779403 10685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:47:06.768252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36195 (local address 127.0.98.62:33313)
0927 12:47:06.768412 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 12:47:06.768416 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:47:06.769292 (+ 876us) server_negotiation.cc:407] Connection header received
0927 12:47:06.770545 (+ 1253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:47:06.770547 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:47:06.770590 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:47:06.770651 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:47:06.771744 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:47:06.772263 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:47:06.773373 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:47:06.773587 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:47:06.773660 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:47:06.775962 (+ 2302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:47:06.775966 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:47:06.775968 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:47:06.775992 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:47:06.777281 (+ 1289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:47:06.777831 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:47:06.777835 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:47:06.777837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:47:06.777918 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:47:06.778285 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:47:06.778288 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:47:06.778290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:47:06.778489 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:47:06.778575 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 12:47:06.779096 (+ 521us) server_negotiation.cc:299] Negotiation successful
0927 12:47:06.779223 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 12:47:06.780200 11107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:47:06.768634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33313 (local address 127.0.98.3:36195)
0927 12:47:06.769135 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0927 12:47:06.769196 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 12:47:06.770368 (+ 1172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:47:06.770786 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:47:06.770792 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:47:06.771134 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:47:06.771629 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:47:06.771642 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:47:06.772359 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:47:06.772361 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:47:06.773281 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:47:06.773290 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:47:06.773710 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:47:06.773714 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:47:06.773855 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:47:06.774423 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:47:06.774439 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:47:06.775809 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:47:06.777416 (+ 1607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:47:06.777423 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:47:06.777439 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:47:06.777734 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:47:06.778001 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:47:06.778004 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:47:06.778006 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:47:06.778196 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:47:06.778585 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:47:06.778592 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:47:06.778856 (+ 264us) client_negotiation.cc:769] Sending connection context
0927 12:47:06.779021 (+ 165us) client_negotiation.cc:240] Negotiation successful
0927 12:47:06.779366 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":124,"threads_started":1}
I20260927 12:47:06.781462 11105 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33313
I20260927 12:47:06.781726 11105 heartbeater.cc:464] Registering TS with master...
I20260927 12:47:06.782367 11105 heartbeater.cc:511] Master 127.0.98.62:33313 requested a full tablet report, sending...
I20260927 12:47:06.783356 10644 ts_manager.cc:194] Registered new tserver with Master: 006df6a8254c4409ac29acf54369e1d1 (127.0.98.3:43049)
I20260927 12:47:06.783929 10644 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36195
I20260927 12:47:06.793996 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 12:47:06.803030 10685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:47:06.795959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33532 (local address 127.0.98.62:33313)
0927 12:47:06.796085 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 12:47:06.796088 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:47:06.796254 (+ 166us) server_negotiation.cc:407] Connection header received
0927 12:47:06.796357 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:47:06.796360 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:47:06.796400 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:47:06.796466 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:47:06.797133 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:47:06.797639 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:47:06.798550 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:47:06.798780 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:47:06.798920 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:47:06.800174 (+ 1254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:47:06.800177 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:47:06.800179 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:47:06.800203 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:47:06.801585 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:47:06.802007 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:47:06.802010 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:47:06.802011 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:47:06.802118 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:47:06.802332 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:47:06.802335 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:47:06.802336 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:47:06.802540 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:47:06.802635 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 12:47:06.802820 (+ 185us) server_negotiation.cc:299] Negotiation successful
0927 12:47:06.802911 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 12:47:06.806126 10644 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33532:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:47:06.809124 10644 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:47:06.824187 11120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:47:06.818228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43049 (local address 127.0.0.1:47228)
0927 12:47:06.819121 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0927 12:47:06.819133 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:47:06.819261 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:47:06.819957 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:47:06.819960 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:47:06.819976 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:47:06.820066 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:47:06.820072 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:47:06.821383 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:47:06.821387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:47:06.822833 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:47:06.822839 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:47:06.823644 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:47:06.823647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:47:06.823757 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:47:06.823794 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 12:47:06.823897 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 12:47:06.824015 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":836,"mutex_wait_us":95,"spinlock_wait_cycles":6016,"thread_start_us":117,"threads_started":1}
I20260927 12:47:06.824365 11121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:47:06.819321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46316 (local address 127.0.98.1:46315)
0927 12:47:06.819706 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0927 12:47:06.819721 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 12:47:06.819925 (+ 204us) server_negotiation.cc:407] Connection header received
0927 12:47:06.820140 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:47:06.820146 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:47:06.820299 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:47:06.820422 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:47:06.820615 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:47:06.821350 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:47:06.822925 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:47:06.823667 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:47:06.823748 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:47:06.823857 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 12:47:06.824132 (+ 275us) server_negotiation.cc:299] Negotiation successful
0927 12:47:06.824233 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":92,"threads_started":1}
I20260927 12:47:06.824397 11117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:47:06.818246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41286 (local address 127.0.98.2:36153)
0927 12:47:06.818890 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0927 12:47:06.818903 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 12:47:06.818928 (+ 25us) server_negotiation.cc:407] Connection header received
0927 12:47:06.819006 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:47:06.819011 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:47:06.819240 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:47:06.819805 (+ 565us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:47:06.819981 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:47:06.821197 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:47:06.822809 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:47:06.823595 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:47:06.823715 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:47:06.823850 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 12:47:06.824067 (+ 217us) server_negotiation.cc:299] Negotiation successful
0927 12:47:06.824166 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":104,"threads_started":1}
I20260927 12:47:06.824190 11118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:47:06.818032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36153 (local address 127.0.0.1:41286)
0927 12:47:06.818609 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0927 12:47:06.818659 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 12:47:06.818782 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:47:06.819436 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:47:06.819442 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:47:06.819468 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:47:06.819626 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:47:06.819634 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:47:06.821329 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:47:06.821332 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:47:06.822702 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:47:06.822711 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:47:06.823850 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:47:06.823853 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:47:06.823941 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:47:06.823960 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 12:47:06.824022 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 12:47:06.824075 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"mutex_wait_us":160,"thread_start_us":83,"threads_started":1}
I20260927 12:47:06.824725 11116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:47:06.818894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46315 (local address 127.0.0.1:46316)
0927 12:47:06.819846 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0927 12:47:06.819859 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:47:06.820002 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:47:06.820408 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:47:06.820411 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:47:06.820428 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:47:06.820521 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:47:06.820527 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:47:06.821465 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:47:06.821468 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:47:06.822811 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:47:06.822823 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:47:06.823914 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:47:06.823917 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:47:06.823991 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:47:06.824013 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 12:47:06.824555 (+ 542us) client_negotiation.cc:240] Negotiation successful
0927 12:47:06.824620 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":895,"mutex_wait_us":7,"spinlock_wait_cycles":1536,"thread_start_us":91,"threads_started":1}
I20260927 12:47:06.825038 11119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:47:06.818719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47228 (local address 127.0.98.3:43049)
0927 12:47:06.819146 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0927 12:47:06.819157 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 12:47:06.819257 (+ 100us) server_negotiation.cc:407] Connection header received
0927 12:47:06.819406 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:47:06.819412 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:47:06.819618 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:47:06.819796 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:47:06.820188 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:47:06.821270 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:47:06.822969 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:47:06.823532 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:47:06.823629 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:47:06.823779 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 12:47:06.824265 (+ 486us) server_negotiation.cc:299] Negotiation successful
0927 12:47:06.824908 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":152,"threads_started":1}
I20260927 12:47:06.827477 11039 tablet_service.cc:1467] Processing CreateTablet for tablet b3787c6ceea04269b228c73816b1b673 (DEFAULT_TABLE table=test-table [id=eccbf34f6ad64c2e879424527c0d141f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:47:06.827656 10771 tablet_service.cc:1467] Processing CreateTablet for tablet b3787c6ceea04269b228c73816b1b673 (DEFAULT_TABLE table=test-table [id=eccbf34f6ad64c2e879424527c0d141f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:47:06.827654 10905 tablet_service.cc:1467] Processing CreateTablet for tablet b3787c6ceea04269b228c73816b1b673 (DEFAULT_TABLE table=test-table [id=eccbf34f6ad64c2e879424527c0d141f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:47:06.828527 11039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3787c6ceea04269b228c73816b1b673. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:47:06.828704 10771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3787c6ceea04269b228c73816b1b673. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:47:06.828725 10905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3787c6ceea04269b228c73816b1b673. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:47:06.834389 11122 tablet_bootstrap.cc:492] T b3787c6ceea04269b228c73816b1b673 P 006df6a8254c4409ac29acf54369e1d1: Bootstrap starting.
I20260927 12:47:06.836278 11123 tablet_bootstrap.cc:492] T b3787c6ceea04269b228c73816b1b673 P 723197480f2e4ac58676b8d5ecacdd4d: Bootstrap starting.
I20260927 12:47:06.837312 11122 tablet_bootstrap.cc:654] T b3787c6ceea04269b228c73816b1b673 P 006df6a8254c4409ac29acf54369e1d1: Neither blocks nor log segments found. Creating new log.
I20260927 12:47:06.838130 11123 tablet_bootstrap.cc:654] T b3787c6ceea04269b228c73816b1b673 P 723197480f2e4ac58676b8d5ecacdd4d: Neither blocks nor log segments found. Creating new log.
I20260927 12:47:06.838116 11122 log.cc:824] T b3787c6ceea04269b228c73816b1b673 P 006df6a8254c4409ac29acf54369e1d1: Log is configured to *not* fsync() on all Append() calls
I20260927 12:47:06.838443 11124 tablet_bootstrap.cc:492] T b3787c6ceea04269b228c73816b1b673 P fe4599b667cc4d299709d9f8acd0bccc: Bootstrap starting.
I20260927 12:47:06.839327 11123 log.cc:824] T b3787c6ceea04269b228c73816b1b673 P 723197480f2e4ac58676b8d5ecacdd4d: Log is configured to *not* fsync() on all Append() calls
I20260927 12:47:06.840039 11122 tablet_bootstrap.cc:492] T b3787c6ceea04269b228c73816b1b673 P 006df6a8254c4409ac29acf54369e1d1: No bootstrap required, opened a new log
I20260927 12:47:06.840224 11122 ts_tablet_manager.cc:1397] T b3787c6ceea04269b228c73816b1b673 P 006df6a8254c4409ac29acf54369e1d1: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 12:47:06.840962 11124 tablet_bootstrap.cc:654] T b3787c6ceea04269b228c73816b1b673 P fe4599b667cc4d299709d9f8acd0bccc: Neither blocks nor log segments found. Creating new log.
I20260927 12:47:06.841395 11123 tablet_bootstrap.cc:492] T b3787c6ceea04269b228c73816b1b673 P 723197480f2e4ac58676b8d5ecacdd4d: No bootstrap required, opened a new log
I20260927 12:47:06.841578 11123 ts_tablet_manager.cc:1397] T b3787c6ceea04269b228c73816b1b673 P 723197480f2e4ac58676b8d5ecacdd4d: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260927 12:47:06.841769 11124 log.cc:824] T b3787c6ceea04269b228c73816b1b673 P fe4599b667cc4d299709d9f8acd0bccc: Log is configured to *not* fsync() on all Append() calls
I20260927 12:47:06.843422 11122 raft_consensus.cc:348] T b3787c6ceea04269b228c73816b1b673 P 006df6a8254c4409ac29acf54369e1d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe4599b667cc4d299709d9f8acd0bccc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36153 } } peers { permanent_uuid: "006df6a8254c4409ac29acf54369e1d1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43049 } } peers { permanent_uuid: "723197480f2e4ac58676b8d5ecacdd4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46315 } }
I20260927 12:47:06.843704 11122 raft_consensus.cc:374] T b3787c6ceea04269b228c73816b1b673 P 006df6a8254c4409ac29acf54369e1d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:47:06.843756 11122 raft_consensus.cc:729] T b3787c6ceea04269b228c73816b1b673 P 006df6a8254c4409ac29acf54369e1d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 006df6a8254c4409ac29acf54369e1d1, State: Initialized, Role: FOLLOWER
I20260927 12:47:06.843762 11124 tablet_bootstrap.cc:492] T b3787c6ceea04269b228c73816b1b673 P fe4599b667cc4d299709d9f8acd0bccc: No bootstrap required, opened a new log
I20260927 12:47:06.843946 11124 ts_tablet_manager.cc:1397] T b3787c6ceea04269b228c73816b1b673 P fe4599b667cc4d299709d9f8acd0bccc: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260927 12:47:06.844260 11122 consensus_queue.cc:260] T b3787c6ceea04269b228c73816b1b673 P 006df6a8254c4409ac29acf54369e1d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe4599b667cc4d299709d9f8acd0bccc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36153 } } peers { permanent_uuid: "006df6a8254c4409ac29acf54369e1d1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43049 } } peers { permanent_uuid: "723197480f2e4ac58676b8d5ecacdd4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46315 } }
I20260927 12:47:06.844877 11105 heartbeater.cc:503] Master 127.0.98.62:33313 was elected leader, sending a full tablet report...
I20260927 12:47:06.845252 11122 ts_tablet_manager.cc:1428] T b3787c6ceea04269b228c73816b1b673 P 006df6a8254c4409ac29acf54369e1d1: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 12:47:06.845158 11123 raft_consensus.cc:348] T b3787c6ceea04269b228c73816b1b673 P 723197480f2e4ac58676b8d5ecacdd4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe4599b667cc4d299709d9f8acd0bccc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36153 } } peers { permanent_uuid: "006df6a8254c4409ac29acf54369e1d1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43049 } } peers { permanent_uuid: "723197480f2e4ac58676b8d5ecacdd4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46315 } }
I20260927 12:47:06.845433 11123 raft_consensus.cc:374] T b3787c6ceea04269b228c73816b1b673 P 723197480f2e4ac58676b8d5ecacdd4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:47:06.845489 11123 raft_consensus.cc:729] T b3787c6ceea04269b228c73816b1b673 P 723197480f2e4ac58676b8d5ecacdd4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 723197480f2e4ac58676b8d5ecacdd4d, State: Initialized, Role: FOLLOWER
I20260927 12:47:06.845947 11123 consensus_queue.cc:260] T b3787c6ceea04269b228c73816b1b673 P 723197480f2e4ac58676b8d5ecacdd4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe4599b667cc4d299709d9f8acd0bccc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36153 } } peers { permanent_uuid: "006df6a8254c4409ac29acf54369e1d1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43049 } } peers { permanent_uuid: "723197480f2e4ac58676b8d5ecacdd4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46315 } }
I20260927 12:47:06.846602 10837 heartbeater.cc:503] Master 127.0.98.62:33313 was elected leader, sending a full tablet report...
I20260927 12:47:06.846941 11123 ts_tablet_manager.cc:1428] T b3787c6ceea04269b228c73816b1b673 P 723197480f2e4ac58676b8d5ecacdd4d: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 12:47:06.847733 11124 raft_consensus.cc:348] T b3787c6ceea04269b228c73816b1b673 P fe4599b667cc4d299709d9f8acd0bccc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe4599b667cc4d299709d9f8acd0bccc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36153 } } peers { permanent_uuid: "006df6a8254c4409ac29acf54369e1d1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43049 } } peers { permanent_uuid: "723197480f2e4ac58676b8d5ecacdd4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46315 } }
I20260927 12:47:06.847952 11124 raft_consensus.cc:374] T b3787c6ceea04269b228c73816b1b673 P fe4599b667cc4d299709d9f8acd0bccc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:47:06.848003 11124 raft_consensus.cc:729] T b3787c6ceea04269b228c73816b1b673 P fe4599b667cc4d299709d9f8acd0bccc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe4599b667cc4d299709d9f8acd0bccc, State: Initialized, Role: FOLLOWER
I20260927 12:47:06.848405 11124 consensus_queue.cc:260] T b3787c6ceea04269b228c73816b1b673 P fe4599b667cc4d299709d9f8acd0bccc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe4599b667cc4d299709d9f8acd0bccc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36153 } } peers { permanent_uuid: "006df6a8254c4409ac29acf54369e1d1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43049 } } peers { permanent_uuid: "723197480f2e4ac58676b8d5ecacdd4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46315 } }
I20260927 12:47:06.849473 11124 ts_tablet_manager.cc:1428] T b3787c6ceea04269b228c73816b1b673 P fe4599b667cc4d299709d9f8acd0bccc: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260927 12:47:06.849793 10971 heartbeater.cc:503] Master 127.0.98.62:33313 was elected leader, sending a full tablet report...
W20260927 12:47:07.017306 11106 tablet.cc:2369] T b3787c6ceea04269b228c73816b1b673 P 006df6a8254c4409ac29acf54369e1d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 12:47:07.044718 10972 tablet.cc:2369] T b3787c6ceea04269b228c73816b1b673 P fe4599b667cc4d299709d9f8acd0bccc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:47:07.047724 11129 raft_consensus.cc:482] T b3787c6ceea04269b228c73816b1b673 P 723197480f2e4ac58676b8d5ecacdd4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:47:07.047925 11129 raft_consensus.cc:504] T b3787c6ceea04269b228c73816b1b673 P 723197480f2e4ac58676b8d5ecacdd4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe4599b667cc4d299709d9f8acd0bccc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36153 } } peers { permanent_uuid: "006df6a8254c4409ac29acf54369e1d1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43049 } } peers { permanent_uuid: "723197480f2e4ac58676b8d5ecacdd4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46315 } }
I20260927 12:47:07.049178 11129 leader_election.cc:288] T b3787c6ceea04269b228c73816b1b673 P 723197480f2e4ac58676b8d5ecacdd4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe4599b667cc4d299709d9f8acd0bccc (127.0.98.2:36153), 006df6a8254c4409ac29acf54369e1d1 (127.0.98.3:43049)
I20260927 12:47:07.054638 11117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:47:07.049525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42209 (local address 127.0.98.2:36153)
0927 12:47:07.049664 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 12:47:07.049668 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:47:07.049920 (+ 252us) server_negotiation.cc:407] Connection header received
0927 12:47:07.050082 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:47:07.050085 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:47:07.050140 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:47:07.050228 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:47:07.050568 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:47:07.051402 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:47:07.053178 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:47:07.053897 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:47:07.053985 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:47:07.054063 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 12:47:07.054416 (+ 353us) server_negotiation.cc:299] Negotiation successful
0927 12:47:07.054502 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 12:47:07.054638 11119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:47:07.049577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43647 (local address 127.0.98.3:43049)
0927 12:47:07.049701 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 12:47:07.049705 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:47:07.049986 (+ 281us) server_negotiation.cc:407] Connection header received
0927 12:47:07.050082 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:47:07.050085 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:47:07.050140 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:47:07.050248 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:47:07.050571 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:47:07.051388 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:47:07.053190 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:47:07.053837 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:47:07.053957 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:47:07.054029 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 12:47:07.054416 (+ 387us) server_negotiation.cc:299] Negotiation successful
0927 12:47:07.054502 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 12:47:07.054683 11132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:47:07.049466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43049 (local address 127.0.98.1:43647)
0927 12:47:07.049877 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0927 12:47:07.049889 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:47:07.049961 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:47:07.050267 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:47:07.050284 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:47:07.050358 (+ 74us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:47:07.050457 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:47:07.050462 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:47:07.051518 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:47:07.051522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:47:07.053044 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:47:07.053055 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:47:07.054015 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:47:07.054019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:47:07.054237 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:47:07.054275 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 12:47:07.054360 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 12:47:07.054477 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"mutex_wait_us":155,"spinlock_wait_cycles":6784,"thread_start_us":93,"threads_started":1}
I20260927 12:47:07.054683 11131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:47:07.049419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36153 (local address 127.0.98.1:42209)
0927 12:47:07.049802 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0927 12:47:07.049817 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:47:07.049959 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:47:07.050267 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:47:07.050270 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:47:07.050293 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:47:07.050439 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:47:07.050447 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:47:07.051518 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:47:07.051522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:47:07.053044 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:47:07.053055 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:47:07.054015 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:47:07.054019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:47:07.054208 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:47:07.054285 (+ 77us) client_negotiation.cc:769] Sending connection context
0927 12:47:07.054360 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 12:47:07.054478 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":197,"thread_start_us":113,"threads_started":1}
I20260927 12:47:07.055491 11059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3787c6ceea04269b228c73816b1b673" candidate_uuid: "723197480f2e4ac58676b8d5ecacdd4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "006df6a8254c4409ac29acf54369e1d1" is_pre_election: true
I20260927 12:47:07.055493 10925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3787c6ceea04269b228c73816b1b673" candidate_uuid: "723197480f2e4ac58676b8d5ecacdd4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe4599b667cc4d299709d9f8acd0bccc" is_pre_election: true
I20260927 12:47:07.055848 11059 raft_consensus.cc:2454] T b3787c6ceea04269b228c73816b1b673 P 006df6a8254c4409ac29acf54369e1d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 723197480f2e4ac58676b8d5ecacdd4d in term 0.
I20260927 12:47:07.055848 10925 raft_consensus.cc:2454] T b3787c6ceea04269b228c73816b1b673 P fe4599b667cc4d299709d9f8acd0bccc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 723197480f2e4ac58676b8d5ecacdd4d in term 0.
I20260927 12:47:07.056352 10725 leader_election.cc:302] T b3787c6ceea04269b228c73816b1b673 P 723197480f2e4ac58676b8d5ecacdd4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 006df6a8254c4409ac29acf54369e1d1, 723197480f2e4ac58676b8d5ecacdd4d; no voters:
I20260927 12:47:07.056641 11129 raft_consensus.cc:2781] T b3787c6ceea04269b228c73816b1b673 P 723197480f2e4ac58676b8d5ecacdd4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:47:07.056731 11129 raft_consensus.cc:482] T b3787c6ceea04269b228c73816b1b673 P 723197480f2e4ac58676b8d5ecacdd4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:47:07.056782 11129 raft_consensus.cc:3037] T b3787c6ceea04269b228c73816b1b673 P 723197480f2e4ac58676b8d5ecacdd4d [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:47:07.057849 11129 raft_consensus.cc:504] T b3787c6ceea04269b228c73816b1b673 P 723197480f2e4ac58676b8d5ecacdd4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe4599b667cc4d299709d9f8acd0bccc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36153 } } peers { permanent_uuid: "006df6a8254c4409ac29acf54369e1d1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43049 } } peers { permanent_uuid: "723197480f2e4ac58676b8d5ecacdd4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46315 } }
I20260927 12:47:07.058261 11129 leader_election.cc:288] T b3787c6ceea04269b228c73816b1b673 P 723197480f2e4ac58676b8d5ecacdd4d [CANDIDATE]: Term 1 election: Requested vote from peers fe4599b667cc4d299709d9f8acd0bccc (127.0.98.2:36153), 006df6a8254c4409ac29acf54369e1d1 (127.0.98.3:43049)
I20260927 12:47:07.058648 10925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3787c6ceea04269b228c73816b1b673" candidate_uuid: "723197480f2e4ac58676b8d5ecacdd4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe4599b667cc4d299709d9f8acd0bccc"
I20260927 12:47:07.058648 11059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3787c6ceea04269b228c73816b1b673" candidate_uuid: "723197480f2e4ac58676b8d5ecacdd4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "006df6a8254c4409ac29acf54369e1d1"
I20260927 12:47:07.058779 11059 raft_consensus.cc:3037] T b3787c6ceea04269b228c73816b1b673 P 006df6a8254c4409ac29acf54369e1d1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:47:07.058779 10925 raft_consensus.cc:3037] T b3787c6ceea04269b228c73816b1b673 P fe4599b667cc4d299709d9f8acd0bccc [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:47:07.060065 10925 raft_consensus.cc:2454] T b3787c6ceea04269b228c73816b1b673 P fe4599b667cc4d299709d9f8acd0bccc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 723197480f2e4ac58676b8d5ecacdd4d in term 1.
I20260927 12:47:07.060065 11059 raft_consensus.cc:2454] T b3787c6ceea04269b228c73816b1b673 P 006df6a8254c4409ac29acf54369e1d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 723197480f2e4ac58676b8d5ecacdd4d in term 1.
I20260927 12:47:07.060382 10724 leader_election.cc:302] T b3787c6ceea04269b228c73816b1b673 P 723197480f2e4ac58676b8d5ecacdd4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 723197480f2e4ac58676b8d5ecacdd4d, fe4599b667cc4d299709d9f8acd0bccc; no voters:
I20260927 12:47:07.060565 11129 raft_consensus.cc:2781] T b3787c6ceea04269b228c73816b1b673 P 723197480f2e4ac58676b8d5ecacdd4d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:47:07.060813 11129 raft_consensus.cc:686] T b3787c6ceea04269b228c73816b1b673 P 723197480f2e4ac58676b8d5ecacdd4d [term 1 LEADER]: Becoming Leader. State: Replica: 723197480f2e4ac58676b8d5ecacdd4d, State: Running, Role: LEADER
I20260927 12:47:07.061177 11129 consensus_queue.cc:237] T b3787c6ceea04269b228c73816b1b673 P 723197480f2e4ac58676b8d5ecacdd4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe4599b667cc4d299709d9f8acd0bccc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36153 } } peers { permanent_uuid: "006df6a8254c4409ac29acf54369e1d1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43049 } } peers { permanent_uuid: "723197480f2e4ac58676b8d5ecacdd4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46315 } }
I20260927 12:47:07.064262 10644 catalog_manager.cc:5495] T b3787c6ceea04269b228c73816b1b673 P 723197480f2e4ac58676b8d5ecacdd4d reported cstate change: term changed from 0 to 1, leader changed from <none> to 723197480f2e4ac58676b8d5ecacdd4d (127.0.98.1). New cstate: current_term: 1 leader_uuid: "723197480f2e4ac58676b8d5ecacdd4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe4599b667cc4d299709d9f8acd0bccc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "006df6a8254c4409ac29acf54369e1d1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "723197480f2e4ac58676b8d5ecacdd4d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46315 } health_report { overall_health: HEALTHY } } }
W20260927 12:47:07.076077 10838 tablet.cc:2369] T b3787c6ceea04269b228c73816b1b673 P 723197480f2e4ac58676b8d5ecacdd4d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:47:07.125849 11121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:47:07.122300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46332 (local address 127.0.98.1:46315)
0927 12:47:07.122412 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 12:47:07.122415 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:47:07.122428 (+ 13us) server_negotiation.cc:407] Connection header received
0927 12:47:07.122504 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:47:07.122506 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:47:07.122560 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:47:07.122650 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:47:07.122972 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:47:07.123488 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:47:07.124371 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:47:07.124621 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:47:07.124729 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:47:07.125080 (+ 351us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:47:07.125527 (+ 447us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:47:07.125638 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 12:47:07.125667 (+ 29us) server_negotiation.cc:299] Negotiation successful
0927 12:47:07.125734 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 12:47:07.132036 11059 raft_consensus.cc:1261] T b3787c6ceea04269b228c73816b1b673 P 006df6a8254c4409ac29acf54369e1d1 [term 1 FOLLOWER]: Refusing update from remote peer 723197480f2e4ac58676b8d5ecacdd4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:47:07.132027 10925 raft_consensus.cc:1261] T b3787c6ceea04269b228c73816b1b673 P fe4599b667cc4d299709d9f8acd0bccc [term 1 FOLLOWER]: Refusing update from remote peer 723197480f2e4ac58676b8d5ecacdd4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:47:07.132836 11133 consensus_queue.cc:1035] T b3787c6ceea04269b228c73816b1b673 P 723197480f2e4ac58676b8d5ecacdd4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe4599b667cc4d299709d9f8acd0bccc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:47:07.133033 11129 consensus_queue.cc:1035] T b3787c6ceea04269b228c73816b1b673 P 723197480f2e4ac58676b8d5ecacdd4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "006df6a8254c4409ac29acf54369e1d1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:47:07.141804 11138 mvcc.cc:205] Tried to move back new op lower bound from 7333942178325377024 to 7333942178045333504. Current Snapshot: MvccSnapshot[applied={T|T < 7333942178325377024 or (T in {7333942178325377024})}]
I20260927 12:47:07.143625 11137 mvcc.cc:205] Tried to move back new op lower bound from 7333942178325377024 to 7333942178045333504. Current Snapshot: MvccSnapshot[applied={T|T < 7333942178325377024}]
I20260927 12:47:07.148101 10646 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33532:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:47:07.148258 10646 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33532:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:47:07.150669 10646 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 11c569487cee436097c31d4f2cfe4c66: Sending DeleteTablet for 3 replicas of tablet b3787c6ceea04269b228c73816b1b673
I20260927 12:47:07.151464 10905 tablet_service.cc:1514] Processing DeleteTablet for tablet b3787c6ceea04269b228c73816b1b673 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:47:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41286
I20260927 12:47:07.151470 11039 tablet_service.cc:1514] Processing DeleteTablet for tablet b3787c6ceea04269b228c73816b1b673 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:47:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47228
I20260927 12:47:07.151631 10771 tablet_service.cc:1514] Processing DeleteTablet for tablet b3787c6ceea04269b228c73816b1b673 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:47:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46316
I20260927 12:47:07.152021 11146 tablet_replica.cc:330] T b3787c6ceea04269b228c73816b1b673 P 006df6a8254c4409ac29acf54369e1d1: stopping tablet replica
I20260927 12:47:07.152125 11148 tablet_replica.cc:330] T b3787c6ceea04269b228c73816b1b673 P 723197480f2e4ac58676b8d5ecacdd4d: stopping tablet replica
I20260927 12:47:07.152431 11148 raft_consensus.cc:2229] T b3787c6ceea04269b228c73816b1b673 P 723197480f2e4ac58676b8d5ecacdd4d [term 1 LEADER]: Raft consensus shutting down.
I20260927 12:47:07.152431 11146 raft_consensus.cc:2229] T b3787c6ceea04269b228c73816b1b673 P 006df6a8254c4409ac29acf54369e1d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:47:07.152827 11148 raft_consensus.cc:2258] T b3787c6ceea04269b228c73816b1b673 P 723197480f2e4ac58676b8d5ecacdd4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:47:07.152832 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 10708
I20260927 12:47:07.152967 11146 raft_consensus.cc:2258] T b3787c6ceea04269b228c73816b1b673 P 006df6a8254c4409ac29acf54369e1d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:47:07.153308 11147 tablet_replica.cc:330] T b3787c6ceea04269b228c73816b1b673 P fe4599b667cc4d299709d9f8acd0bccc: stopping tablet replica
I20260927 12:47:07.153579 11147 raft_consensus.cc:2229] T b3787c6ceea04269b228c73816b1b673 P fe4599b667cc4d299709d9f8acd0bccc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:47:07.153839 11147 raft_consensus.cc:2258] T b3787c6ceea04269b228c73816b1b673 P fe4599b667cc4d299709d9f8acd0bccc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:47:07.155169 11147 ts_tablet_manager.cc:1905] T b3787c6ceea04269b228c73816b1b673 P fe4599b667cc4d299709d9f8acd0bccc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:47:07.157505 11147 ts_tablet_manager.cc:1918] T b3787c6ceea04269b228c73816b1b673 P fe4599b667cc4d299709d9f8acd0bccc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:47:07.157593 11147 log.cc:1196] T b3787c6ceea04269b228c73816b1b673 P fe4599b667cc4d299709d9f8acd0bccc: Deleting WAL directory at /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/minicluster-data/ts-1/wal/wals/b3787c6ceea04269b228c73816b1b673
I20260927 12:47:07.157912 11147 ts_tablet_manager.cc:1939] T b3787c6ceea04269b228c73816b1b673 P fe4599b667cc4d299709d9f8acd0bccc: Deleting consensus metadata
I20260927 12:47:07.158812 11146 ts_tablet_manager.cc:1905] T b3787c6ceea04269b228c73816b1b673 P 006df6a8254c4409ac29acf54369e1d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:47:07.159303 10630 catalog_manager.cc:4841] TS fe4599b667cc4d299709d9f8acd0bccc (127.0.98.2:36153): tablet b3787c6ceea04269b228c73816b1b673 (table test-table [id=eccbf34f6ad64c2e879424527c0d141f]) successfully deleted
W20260927 12:47:07.160316 10633 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46315 (error 108)
I20260927 12:47:07.161128 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 10842
I20260927 12:47:07.161361 11116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:47:07.161056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46315 (local address 127.0.0.1:46346)
0927 12:47:07.161200 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 12:47:07.161273 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 12:47:07.161245 11146 ts_tablet_manager.cc:1918] T b3787c6ceea04269b228c73816b1b673 P 006df6a8254c4409ac29acf54369e1d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:47:07.161985 11146 log.cc:1196] T b3787c6ceea04269b228c73816b1b673 P 006df6a8254c4409ac29acf54369e1d1: Deleting WAL directory at /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790513161160093-392-0/minicluster-data/ts-2/wal/wals/b3787c6ceea04269b228c73816b1b673
I20260927 12:47:07.162320 11146 ts_tablet_manager.cc:1939] T b3787c6ceea04269b228c73816b1b673 P 006df6a8254c4409ac29acf54369e1d1: Deleting consensus metadata
W20260927 12:47:07.162738 10633 catalog_manager.cc:4584] TS 723197480f2e4ac58676b8d5ecacdd4d (127.0.98.1:46315): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b3787c6ceea04269b228c73816b1b673: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46315: connect: Connection refused (error 111)
I20260927 12:47:07.163492 10631 catalog_manager.cc:4841] TS 006df6a8254c4409ac29acf54369e1d1 (127.0.98.3:43049): tablet b3787c6ceea04269b228c73816b1b673 (table test-table [id=eccbf34f6ad64c2e879424527c0d141f]) successfully deleted
I20260927 12:47:07.165934 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 10976
I20260927 12:47:07.170583 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 10614
2026-09-27T12:47:07Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2431 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:47:07 dist-test-slave-lkn4 krb5kdc[11152](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:47:07 dist-test-slave-lkn4 krb5kdc[11152](info): set up 2 sockets
Sep 27 12:47:07 dist-test-slave-lkn4 krb5kdc[11152](info): commencing operation
krb5kdc: starting...
W20260927 12:47:08.134945 392 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/krb5kdc/test-user.keytab.
Sep 27 12:47:08 dist-test-slave-lkn4 krb5kdc[11152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513228, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:47:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:47:08Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260927 12:47:08.300519 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskTC8Et9/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/postgres -l logfile start
2026-09-27 12:47:10.802 UTC [11182] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 12:47:10.802 UTC [11182] LOG: listening on IPv4 address "127.0.98.20", port 54539
2026-09-27 12:47:10.806 UTC [11182] LOG: listening on Unix socket "/tmp/.s.PGSQL.54539"
2026-09-27 12:47:10.828 UTC [11185] LOG: database system was shut down at 2026-09-27 12:47:09 UTC
2026-09-27 12:47:10.831 UTC [11182] LOG: database system is ready to accept connections
I20260927 12:47:11.710672 392 mini_postgres.cc:96] Postgres bound to 54539
2026-09-27 12:47:11.716 UTC [11193] FATAL: database "slave" does not exist
127.0.98.20:54539 - accepting connections
I20260927 12:47:11.717581 392 mini_ranger.cc:161] Starting Ranger...
I20260927 12:47:11.731834 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 12:47:12.202255 392 mini_ranger.cc:87] Created ranger Postgres database
I20260927 12:47:12.202342 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/ranger-admin
2026-09-27 12:47:12,475 [I] DB FLAVOR :POSTGRES
2026-09-27 12:47:12,476 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 12:47:12,476 [I] Checking connection..
2026-09-27 12:47:12,476 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTC8Et9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTC8Et9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 12:47:12,716 [I] Checking connection passed.
2026-09-27 12:47:12,716 [I] --------- Verifying version history table ---------
2026-09-27 12:47:12,716 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTC8Et9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTC8Et9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 12:47:12,981 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 12:47:12,982 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTC8Et9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTC8Et9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 12:47:13,239 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 12:47:13,239 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 12:47:13,239 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTC8Et9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTC8Et9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54539/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskTC8Et9/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 12:47:13.490 UTC [11257] WARNING: there is no transaction in progress
2026-09-27 12:47:13,496 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 12:47:13,496 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTC8Et9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTC8Et9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 12:47:13,725 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 12:47:13,726 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 12:47:13,726 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTC8Et9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTC8Et9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 12:47:13,966 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTC8Et9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTC8Et9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 12:47:14,215 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTC8Et9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTC8Et9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-lkn4.c.gcp-upstream.internal','N') ;"
2026-09-27 12:47:14,468 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTC8Et9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTC8Et9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 12:47:14,707 [I] Table x_portal_user does not exist in database ranger
2026-09-27 12:47:14,707 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTC8Et9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTC8Et9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 12:47:14,936 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 12:47:14,936 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTC8Et9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTC8Et9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 12:47:15,189 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTC8Et9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTC8Et9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 12:47:15,441 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 12:47:15,441 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTC8Et9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTC8Et9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54539/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskTC8Et9/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 12:47:16.438 UTC [11365] WARNING: there is no transaction in progress
2026-09-27 12:47:16.456 UTC [11365] WARNING: there is no transaction in progress
2026-09-27 12:47:16.474 UTC [11365] WARNING: there is no transaction in progress
2026-09-27 12:47:16.490 UTC [11365] WARNING: there is no transaction in progress
2026-09-27 12:47:16.508 UTC [11365] WARNING: there is no transaction in progress
2026-09-27 12:47:16.526 UTC [11365] WARNING: there is no transaction in progress
2026-09-27 12:47:16.586 UTC [11365] WARNING: there is no transaction in progress
2026-09-27 12:47:16.594 UTC [11365] WARNING: there is no transaction in progress
2026-09-27 12:47:16.609 UTC [11365] WARNING: there is no transaction in progress
2026-09-27 12:47:16.620 UTC [11365] WARNING: there is no transaction in progress
2026-09-27 12:47:16.632 UTC [11365] WARNING: there is no transaction in progress
2026-09-27 12:47:16.646 UTC [11365] WARNING: there is no transaction in progress
2026-09-27 12:47:16.657 UTC [11365] WARNING: there is no transaction in progress
2026-09-27 12:47:16.664 UTC [11365] WARNING: there is no transaction in progress
2026-09-27 12:47:16.672 UTC [11365] WARNING: there is no transaction in progress
2026-09-27 12:47:16.679 UTC [11365] WARNING: there is no transaction in progress
2026-09-27 12:47:17,362 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 12:47:17,363 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTC8Et9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTC8Et9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 12:47:17,590 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 12:47:17,590 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTC8Et9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTC8Et9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 12:47:17,834 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 12:47:17,834 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTC8Et9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTC8Et9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 12:47:18,072 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 12:47:18,072 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTC8Et9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTC8Et9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 12:47:18,315 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTC8Et9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTC8Et9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 12:47:18,548 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTC8Et9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTC8Et9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-lkn4.c.gcp-upstream.internal';"
2026-09-27 12:47:18,798 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 12:47:18,798 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTC8Et9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTC8Et9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54539/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 12:47:19,056 [I] DB_PATCHES have already been applied
I20260927 12:47:19.062002 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskTC8Et9/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskTC8Et9/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskTC8Et9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTC8Et9/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 12:47:19.062090 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260927 12:47:19.065358 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:36449
Sep 27, 2026 12:47:19 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 12:47:19 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 12:47:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 12:47:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 12:47:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 12:47:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 12:47:20 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 12:47:20.080283 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.363s user 0.000s sys 0.009s
Sep 27, 2026 12:47:40 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 12:47:40 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 27, 2026 12:47:40 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 12:47:40 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 12:47:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 12:47:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 12:47:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 12:47:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 12:47:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 12:47:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 12:47:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 12:47:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 12:47:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 12:47:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 12:47:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 12:47:40 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 12:47:40 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 12:47:41 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 12:47:42.076073 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 12:47:42 dist-test-slave-lkn4 krb5kdc[11152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513262, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 12:47:42.157425 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 12:47:42.166424 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 12:47:43.097012 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 12:47:43.097114 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/ranger-kms
I20260927 12:47:43.097208 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:54539
2026-09-27 12:47:43,268 [I] DB FLAVOR :POSTGRES
2026-09-27 12:47:43,269 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 12:47:43,269 [I] Checking connection
2026-09-27 12:47:43,269 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTC8Et9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54539/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 12:47:43,530 [I] connection success
2026-09-27 12:47:43,530 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 12:47:43,530 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 12:47:43,530 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTC8Et9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54539/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 12:47:43,762 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 12:47:43,762 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 12:47:43,762 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 12:47:43,762 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTC8Et9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:54539/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 12:47:44,016 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 12:47:44.484423 392 mini_ranger_kms.cc:310] Created kms service
I20260927 12:47:44.664412 392 mini_ranger_kms.cc:326] Created kudu user
I20260927 12:47:44.711166 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 12:47:44 dist-test-slave-lkn4 krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513262, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 12:47:45.097856 392 mini_ranger_kms.cc:384] Added ranger policy
I20260927 12:47:45.097994 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskTC8Et9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTC8Et9/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskTC8Et9/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 12:47:45.098023 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260927 12:47:45.101564 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11545
I20260927 12:47:45.101704 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:45649
Sep 27, 2026 12:47:46 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 12:47:46 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 12:47:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 12:47:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 12:47:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 12:47:46 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 12:47:47.178344 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.021s user 0.015s sys 0.139s
I20260927 12:47:47.178478 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 12:47:47.178606 392 mini_ranger_kms.cc:401] 127.0.98.20:45649/kms/v1/keys
Sep 27 12:47:47 dist-test-slave-lkn4 krb5kdc[11152](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790513267, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 12:47:47 dist-test-slave-lkn4 krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513267, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 12:47:48 dist-test-slave-lkn4 krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513267, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 12:47:50 dist-test-slave-lkn4 krb5kdc[11152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:47:50.368543 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43913
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43913
--ranger_config_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:45649/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:47:50.481379 11596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:47:50.481670 11596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:47:50.481709 11596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:47:50.485951 11596 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:47:50.486027 11596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:47:50.486070 11596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:47:50.486092 11596 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:47:50.486114 11596 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:47:50.491837 11596 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:45649/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43913
--ranger_config_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:47:50.492980 11596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:47:50.494328 11596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:47:50.503448 11601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:47:50.503530 11602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:47:50.503613 11604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:47:50.503676 11596 server_base.cc:1034] running on GCE node
I20260927 12:47:50.504436 11596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:47:50.505609 11596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:47:50.506775 11596 hybrid_clock.cc:648] HybridClock initialized: now 1790513270506753 us; error 41 us; skew 500 ppm
Sep 27 12:47:50 dist-test-slave-lkn4 krb5kdc[11152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513270, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:47:50.509445 11596 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:47:50.510435 11596 webserver.cc:460] Webserver started at http://127.0.98.62:34325/ using document root <none> and password file <none>
I20260927 12:47:50.510967 11596 fs_manager.cc:359] Metadata directory not provided
I20260927 12:47:50.511020 11596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:47:50.511215 11596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 12:47:50 dist-test-slave-lkn4 krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513270, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 12:47:50.575955 11596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/master-0/data/instance:
uuid: "0a02703ac1334a629da126a949285530"
format_stamp: "Formatted at 2026-09-27 12:47:50 on dist-test-slave-lkn4"
server_key: "81306c2d4c1281c75ff3727f9c19a7f6"
server_key_iv: "76551b73026dd443694f2b02cd0819d2"
server_key_version: "kuduclusterkey@0"
I20260927 12:47:50.576623 11596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/master-0/wal/instance:
uuid: "0a02703ac1334a629da126a949285530"
format_stamp: "Formatted at 2026-09-27 12:47:50 on dist-test-slave-lkn4"
server_key: "81306c2d4c1281c75ff3727f9c19a7f6"
server_key_iv: "76551b73026dd443694f2b02cd0819d2"
server_key_version: "kuduclusterkey@0"
I20260927 12:47:50.580502 11596 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 12:47:50.598744 11611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:47:50.600147 11596 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 12:47:50.600271 11596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/master-0/wal
uuid: "0a02703ac1334a629da126a949285530"
format_stamp: "Formatted at 2026-09-27 12:47:50 on dist-test-slave-lkn4"
server_key: "81306c2d4c1281c75ff3727f9c19a7f6"
server_key_iv: "76551b73026dd443694f2b02cd0819d2"
server_key_version: "kuduclusterkey@0"
I20260927 12:47:50.600401 11596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:47:50.616914 11596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:47:50.619778 11596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:47:50.619954 11596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:47:50.628123 11596 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43913
I20260927 12:47:50.628151 11663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43913 every 8 connection(s)
I20260927 12:47:50.629110 11596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:47:50.631942 11664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:47:50.636242 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 11596
I20260927 12:47:50.636377 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/master-0/wal/instance
I20260927 12:47:50.636673 392 external_mini_cluster.cc:1428] Setting key ab1a46076638abed75d95855b6338ddc
I20260927 12:47:50.638159 11664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a02703ac1334a629da126a949285530: Bootstrap starting.
I20260927 12:47:50.640952 11664 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a02703ac1334a629da126a949285530: Neither blocks nor log segments found. Creating new log.
I20260927 12:47:50.641824 11664 log.cc:824] T 00000000000000000000000000000000 P 0a02703ac1334a629da126a949285530: Log is configured to *not* fsync() on all Append() calls
I20260927 12:47:50.643893 11664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a02703ac1334a629da126a949285530: No bootstrap required, opened a new log
Sep 27 12:47:50 dist-test-slave-lkn4 krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:47:50.647727 11664 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a02703ac1334a629da126a949285530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a02703ac1334a629da126a949285530" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43913 } }
I20260927 12:47:50.648000 11664 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a02703ac1334a629da126a949285530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:47:50.648087 11664 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a02703ac1334a629da126a949285530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a02703ac1334a629da126a949285530, State: Initialized, Role: FOLLOWER
I20260927 12:47:50.648659 11664 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a02703ac1334a629da126a949285530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a02703ac1334a629da126a949285530" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43913 } }
I20260927 12:47:50.648818 11664 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a02703ac1334a629da126a949285530 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:47:50.648906 11664 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a02703ac1334a629da126a949285530 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:47:50.649003 11664 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a02703ac1334a629da126a949285530 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:47:50.650125 11664 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a02703ac1334a629da126a949285530 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a02703ac1334a629da126a949285530" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43913 } }
I20260927 12:47:50.650476 11664 leader_election.cc:302] T 00000000000000000000000000000000 P 0a02703ac1334a629da126a949285530 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a02703ac1334a629da126a949285530; no voters:
I20260927 12:47:50.650826 11664 leader_election.cc:288] T 00000000000000000000000000000000 P 0a02703ac1334a629da126a949285530 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:47:50.650906 11669 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a02703ac1334a629da126a949285530 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:47:50.651166 11669 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a02703ac1334a629da126a949285530 [term 1 LEADER]: Becoming Leader. State: Replica: 0a02703ac1334a629da126a949285530, State: Running, Role: LEADER
I20260927 12:47:50.651510 11669 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a02703ac1334a629da126a949285530 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a02703ac1334a629da126a949285530" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43913 } }
I20260927 12:47:50.652251 11664 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a02703ac1334a629da126a949285530 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:47:50.652915 11669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a02703ac1334a629da126a949285530 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a02703ac1334a629da126a949285530. Latest consensus state: current_term: 1 leader_uuid: "0a02703ac1334a629da126a949285530" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a02703ac1334a629da126a949285530" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43913 } } }
I20260927 12:47:50.653110 11669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a02703ac1334a629da126a949285530 [sys.catalog]: This master's current role is: LEADER
I20260927 12:47:50.653343 11669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a02703ac1334a629da126a949285530 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a02703ac1334a629da126a949285530" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a02703ac1334a629da126a949285530" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43913 } } }
I20260927 12:47:50.653411 11667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:47:50.638384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40700 (local address 127.0.98.62:43913)
0927 12:47:50.638854 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0927 12:47:50.638873 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 12:47:50.638916 (+ 43us) server_negotiation.cc:407] Connection header received
0927 12:47:50.639789 (+ 873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:47:50.639814 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:47:50.640242 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:47:50.640626 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:47:50.641489 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:47:50.642829 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:47:50.644055 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:47:50.644769 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:47:50.644922 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:47:50.647859 (+ 2937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:47:50.647866 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:47:50.647882 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:47:50.647914 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:47:50.649945 (+ 2031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:47:50.650449 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:47:50.650455 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:47:50.650461 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:47:50.650619 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:47:50.650946 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:47:50.650949 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:47:50.650951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:47:50.651519 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:47:50.651693 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 12:47:50.652123 (+ 430us) server_negotiation.cc:299] Negotiation successful
0927 12:47:50.652448 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":150,"threads_started":1}
I20260927 12:47:50.654500 11664 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 12:47:50.655207 11669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a02703ac1334a629da126a949285530 [sys.catalog]: This master's current role is: LEADER
I20260927 12:47:50.656472 11678 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:47:50.662345 11678 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:47:50.673377 11678 catalog_manager.cc:1347] Generated new cluster ID: ada38f0a58ae4541acd339aa4780da90
I20260927 12:47:50.673533 11678 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:47:50.682332 11678 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:47:50.683636 11678 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:47:50.692883 11678 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a02703ac1334a629da126a949285530: Generated new TSK 0
I20260927 12:47:50.693876 11678 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 12:47:51.393303 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:45649/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43913
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:47:51.509832 11709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:47:51.510134 11709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:47:51.510197 11709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:47:51.514847 11709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:47:51.514918 11709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:47:51.514998 11709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:47:51.520637 11709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:45649/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43913
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:47:51.521740 11709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:47:51.522821 11709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:47:51.532928 11714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:47:51.533685 11709 server_base.cc:1034] running on GCE node
W20260927 12:47:51.533761 11715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:47:51.534346 11717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:47:51.534823 11709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:47:51.535513 11709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:47:51.537026 11709 hybrid_clock.cc:648] HybridClock initialized: now 1790513271536979 us; error 63 us; skew 500 ppm
Sep 27 12:47:51 dist-test-slave-lkn4 krb5kdc[11152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:47:51.545812 11709 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 12:47:51.547066 11709 webserver.cc:460] Webserver started at http://127.0.98.1:34729/ using document root <none> and password file <none>
I20260927 12:47:51.547914 11709 fs_manager.cc:359] Metadata directory not provided
I20260927 12:47:51.548060 11709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:47:51.548368 11709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 12:47:51 dist-test-slave-lkn4 krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 12:47:51.566123 11709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/ts-0/data/instance:
uuid: "bb18193f93d6478191bffcf0d6700b09"
format_stamp: "Formatted at 2026-09-27 12:47:51 on dist-test-slave-lkn4"
server_key: "586d9ca70af848f69cf804435550bab9"
server_key_iv: "733faa4170c3ee38cf3b980068fdb9f9"
server_key_version: "kuduclusterkey@0"
I20260927 12:47:51.566905 11709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/ts-0/wal/instance:
uuid: "bb18193f93d6478191bffcf0d6700b09"
format_stamp: "Formatted at 2026-09-27 12:47:51 on dist-test-slave-lkn4"
server_key: "586d9ca70af848f69cf804435550bab9"
server_key_iv: "733faa4170c3ee38cf3b980068fdb9f9"
server_key_version: "kuduclusterkey@0"
I20260927 12:47:51.575553 11709 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.004s sys 0.000s
I20260927 12:47:51.595788 11726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:47:51.597540 11709 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260927 12:47:51.597684 11709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/ts-0/wal
uuid: "bb18193f93d6478191bffcf0d6700b09"
format_stamp: "Formatted at 2026-09-27 12:47:51 on dist-test-slave-lkn4"
server_key: "586d9ca70af848f69cf804435550bab9"
server_key_iv: "733faa4170c3ee38cf3b980068fdb9f9"
server_key_version: "kuduclusterkey@0"
I20260927 12:47:51.597837 11709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:47:51.647167 11709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:47:51.650441 11709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:47:51.650655 11709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:47:51.651450 11709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:47:51.652693 11709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:47:51.652763 11709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:47:51.652823 11709 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:47:51.652904 11709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260927 12:47:51.668294 11709 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42893
I20260927 12:47:51.670171 11709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:47:51.672838 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 11709
I20260927 12:47:51.672964 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:47:51.673270 392 external_mini_cluster.cc:1428] Setting key 7247b68d20d262dcb6d22e697f7a9093
I20260927 12:47:51.671983 11841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42893 every 8 connection(s)
Sep 27 12:47:51 dist-test-slave-lkn4 krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 12:47:51.694222 11844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:47:51.675000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43913 (local address 127.0.98.1:43693)
0927 12:47:51.678308 (+ 3308us) negotiation.cc:107] Waiting for socket to connect
0927 12:47:51.678353 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 12:47:51.679175 (+ 822us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:47:51.679857 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:47:51.679868 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:47:51.680249 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:47:51.680942 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:47:51.680962 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:47:51.682154 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:47:51.682157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:47:51.683553 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:47:51.683565 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:47:51.684412 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:47:51.684417 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:47:51.684642 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:47:51.685408 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:47:51.685431 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:47:51.687381 (+ 1950us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:47:51.689913 (+ 2532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:47:51.689921 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:47:51.689943 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:47:51.690374 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:47:51.690739 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:47:51.690742 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:47:51.690744 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:47:51.690958 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:47:51.691596 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:47:51.691604 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:47:51.692002 (+ 398us) client_negotiation.cc:769] Sending connection context
0927 12:47:51.692272 (+ 270us) client_negotiation.cc:240] Negotiation successful
0927 12:47:51.692674 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3101,"thread_start_us":187,"threads_started":1}
I20260927 12:47:51.694710 11845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:47:51.676168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43693 (local address 127.0.98.62:43913)
0927 12:47:51.676589 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0927 12:47:51.676601 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 12:47:51.678461 (+ 1860us) server_negotiation.cc:407] Connection header received
0927 12:47:51.679384 (+ 923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:47:51.679387 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:47:51.679445 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:47:51.679538 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:47:51.681118 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:47:51.681969 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:47:51.683698 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:47:51.684079 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:47:51.684212 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:47:51.687625 (+ 3413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:47:51.687630 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:47:51.687633 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:47:51.687668 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:47:51.689741 (+ 2073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:47:51.690507 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:47:51.690511 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:47:51.690513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:47:51.690621 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:47:51.691216 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:47:51.691219 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:47:51.691221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:47:51.691477 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:47:51.691582 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 12:47:51.692349 (+ 767us) server_negotiation.cc:299] Negotiation successful
0927 12:47:51.694539 (+ 2190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":90,"threads_started":1}
I20260927 12:47:51.696009 11842 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43913
I20260927 12:47:51.696290 11842 heartbeater.cc:464] Registering TS with master...
I20260927 12:47:51.696951 11842 heartbeater.cc:511] Master 127.0.98.62:43913 requested a full tablet report, sending...
I20260927 12:47:51.699059 11628 ts_manager.cc:194] Registered new tserver with Master: bb18193f93d6478191bffcf0d6700b09 (127.0.98.1:42893)
I20260927 12:47:51.700632 11628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43693
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 12:47:51.711479 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:45649/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43913
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 12:47:51 dist-test-slave-lkn4 krb5kdc[11152](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790513271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 12:47:51.849789 11848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:47:51.850113 11848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:47:51.850212 11848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:47:51.854671 11848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:47:51.854738 11848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:47:51.854818 11848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:47:51.860559 11848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:45649/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43913
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:47:51.861662 11848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:47:51.862752 11848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:47:51.872483 11854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:47:51.872619 11848 server_base.cc:1034] running on GCE node
W20260927 12:47:51.872828 11857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:47:51.873065 11855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:47:51.873533 11848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:47:51.874274 11848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:47:51.875484 11848 hybrid_clock.cc:648] HybridClock initialized: now 1790513271875452 us; error 43 us; skew 500 ppm
Sep 27 12:47:51 dist-test-slave-lkn4 krb5kdc[11152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:47:51.878590 11848 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 12:47:51.879774 11848 webserver.cc:460] Webserver started at http://127.0.98.2:45745/ using document root <none> and password file <none>
I20260927 12:47:51.880548 11848 fs_manager.cc:359] Metadata directory not provided
I20260927 12:47:51.880697 11848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:47:51.880990 11848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 12:47:51 dist-test-slave-lkn4 krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 12:47:51.898950 11848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/ts-1/data/instance:
uuid: "515cff1da7a946da90528a90c91540d4"
format_stamp: "Formatted at 2026-09-27 12:47:51 on dist-test-slave-lkn4"
server_key: "e3faf28fcf1d821a4b98d1fad44317e0"
server_key_iv: "f083f75dc0de3040874952e98053438e"
server_key_version: "kuduclusterkey@0"
I20260927 12:47:51.899909 11848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/ts-1/wal/instance:
uuid: "515cff1da7a946da90528a90c91540d4"
format_stamp: "Formatted at 2026-09-27 12:47:51 on dist-test-slave-lkn4"
server_key: "e3faf28fcf1d821a4b98d1fad44317e0"
server_key_iv: "f083f75dc0de3040874952e98053438e"
server_key_version: "kuduclusterkey@0"
I20260927 12:47:51.905357 11848 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 12:47:51.923111 11864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:47:51.924903 11848 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 12:47:51.925050 11848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/ts-1/wal
uuid: "515cff1da7a946da90528a90c91540d4"
format_stamp: "Formatted at 2026-09-27 12:47:51 on dist-test-slave-lkn4"
server_key: "e3faf28fcf1d821a4b98d1fad44317e0"
server_key_iv: "f083f75dc0de3040874952e98053438e"
server_key_version: "kuduclusterkey@0"
I20260927 12:47:51.925208 11848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:47:51.951151 11848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:47:51.955005 11848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:47:51.955261 11848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:47:51.956003 11848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:47:51.957134 11848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:47:51.957248 11848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:47:51.957351 11848 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:47:51.957412 11848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:47:51.976748 11848 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41877
I20260927 12:47:51.978020 11848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:47:51.979358 11977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41877 every 8 connection(s)
I20260927 12:47:51.981302 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 11848
I20260927 12:47:51.981419 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:47:51.981752 392 external_mini_cluster.cc:1428] Setting key c9d0d8a5e537a83061b2fbd0fe693dca
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Sep 27 12:47:52 dist-test-slave-lkn4 krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:47:52.007507 11845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:47:51.987461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57341 (local address 127.0.98.62:43913)
0927 12:47:51.987582 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 12:47:51.987587 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:47:51.988998 (+ 1411us) server_negotiation.cc:407] Connection header received
0927 12:47:51.990152 (+ 1154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:47:51.990156 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:47:51.990210 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:47:51.990289 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:47:51.996727 (+ 6438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:47:51.997593 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:47:51.999437 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:47:51.999789 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:47:51.999871 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:47:52.003079 (+ 3208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:47:52.003083 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:47:52.003086 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:47:52.003118 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:47:52.004837 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:47:52.005504 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:47:52.005507 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:47:52.005509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:47:52.005592 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:47:52.005992 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:47:52.005995 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:47:52.005997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:47:52.006244 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:47:52.006324 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 12:47:52.007185 (+ 861us) server_negotiation.cc:299] Negotiation successful
0927 12:47:52.007318 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 12:47:52.011574 11981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:47:51.986364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43913 (local address 127.0.98.2:57341)
0927 12:47:51.988849 (+ 2485us) negotiation.cc:107] Waiting for socket to connect
0927 12:47:51.988905 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 12:47:51.989906 (+ 1001us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:47:51.990645 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:47:51.990655 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:47:51.991004 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:47:51.991803 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:47:51.991825 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:47:51.997792 (+ 5967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:47:51.997796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:47:51.999297 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:47:51.999310 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:47:52.000095 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:47:52.000099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:47:52.000308 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:47:52.000953 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:47:52.000975 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:47:52.002872 (+ 1897us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:47:52.005022 (+ 2150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:47:52.005029 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:47:52.005047 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:47:52.005390 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:47:52.005705 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:47:52.005708 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:47:52.005710 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:47:52.005890 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:47:52.006390 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:47:52.006397 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:47:52.006770 (+ 373us) client_negotiation.cc:769] Sending connection context
0927 12:47:52.008071 (+ 1301us) client_negotiation.cc:240] Negotiation successful
0927 12:47:52.008441 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2257,"thread_start_us":142,"threads_started":1}
I20260927 12:47:52.013183 11978 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43913
I20260927 12:47:52.013438 11978 heartbeater.cc:464] Registering TS with master...
I20260927 12:47:52.014016 11978 heartbeater.cc:511] Master 127.0.98.62:43913 requested a full tablet report, sending...
I20260927 12:47:52.015228 11628 ts_manager.cc:194] Registered new tserver with Master: 515cff1da7a946da90528a90c91540d4 (127.0.98.2:41877)
I20260927 12:47:52.015954 11628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57341
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 12:47:52.033288 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:45649/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43913
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:47:52.197213 11983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:47:52.197674 11983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:47:52.197791 11983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:47:52.203867 11983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:47:52.204013 11983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:47:52.204152 11983 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:47:52.212327 11983 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:45649/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43913
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:47:52.214079 11983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:47:52.215566 11983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Sep 27 12:47:52 dist-test-slave-lkn4 krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 12:47:52.229846 11983 server_base.cc:1034] running on GCE node
W20260927 12:47:52.230595 11991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:47:52.230602 11989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:47:52.231060 11988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:47:52.231621 11983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:47:52.232441 11983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:47:52.234184 11983 hybrid_clock.cc:648] HybridClock initialized: now 1790513272234114 us; error 87 us; skew 500 ppm
Sep 27 12:47:52 dist-test-slave-lkn4 krb5kdc[11152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513272, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:47:52.237561 11983 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 12:47:52.238772 11983 webserver.cc:460] Webserver started at http://127.0.98.3:35899/ using document root <none> and password file <none>
I20260927 12:47:52.239530 11983 fs_manager.cc:359] Metadata directory not provided
I20260927 12:47:52.239666 11983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:47:52.239943 11983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 12:47:52 dist-test-slave-lkn4 krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513272, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 12:47:52.259001 11983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/ts-2/data/instance:
uuid: "1bbd576cc38544f68695229667c560e0"
format_stamp: "Formatted at 2026-09-27 12:47:52 on dist-test-slave-lkn4"
server_key: "7718018d0b2bf545763ef21116bf7427"
server_key_iv: "ffd25c7430d24f83ddc251930e9c097c"
server_key_version: "kuduclusterkey@0"
I20260927 12:47:52.259843 11983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1bbd576cc38544f68695229667c560e0"
format_stamp: "Formatted at 2026-09-27 12:47:52 on dist-test-slave-lkn4"
server_key: "7718018d0b2bf545763ef21116bf7427"
server_key_iv: "ffd25c7430d24f83ddc251930e9c097c"
server_key_version: "kuduclusterkey@0"
I20260927 12:47:52.278463 11983 fs_manager.cc:691] Time spent creating directory manager: real 0.018s user 0.004s sys 0.000s
I20260927 12:47:52.309376 11998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:47:52.311360 11983 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.002s sys 0.001s
I20260927 12:47:52.311510 11983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/ts-2/wal
uuid: "1bbd576cc38544f68695229667c560e0"
format_stamp: "Formatted at 2026-09-27 12:47:52 on dist-test-slave-lkn4"
server_key: "7718018d0b2bf545763ef21116bf7427"
server_key_iv: "ffd25c7430d24f83ddc251930e9c097c"
server_key_version: "kuduclusterkey@0"
I20260927 12:47:52.311678 11983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:47:52.353948 11983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:47:52.357756 11983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:47:52.358017 11983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:47:52.358852 11983 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:47:52.359975 11983 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:47:52.360091 11983 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:47:52.360183 11983 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:47:52.360244 11983 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:47:52.369940 11983 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45913
I20260927 12:47:52.372440 11983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:47:52.373208 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 11983
I20260927 12:47:52.373323 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:47:52.373584 392 external_mini_cluster.cc:1428] Setting key 5d322ba72101df6f5c14d83b3c955e0d
I20260927 12:47:52.380110 12111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45913 every 8 connection(s)
Sep 27 12:47:52 dist-test-slave-lkn4 krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513272, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:47:52.397034 11845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:47:52.382482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47085 (local address 127.0.98.62:43913)
0927 12:47:52.382622 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 12:47:52.382626 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:47:52.383861 (+ 1235us) server_negotiation.cc:407] Connection header received
0927 12:47:52.385077 (+ 1216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:47:52.385081 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:47:52.385133 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:47:52.385219 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:47:52.386669 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:47:52.387470 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:47:52.389109 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:47:52.389447 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:47:52.389565 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:47:52.392637 (+ 3072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:47:52.392641 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:47:52.392644 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:47:52.392760 (+ 116us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:47:52.394477 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:47:52.395105 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:47:52.395109 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:47:52.395110 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:47:52.395201 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:47:52.395595 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:47:52.395598 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:47:52.395600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:47:52.395829 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:47:52.395924 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 12:47:52.396742 (+ 818us) server_negotiation.cc:299] Negotiation successful
0927 12:47:52.396871 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 12:47:52.398716 12114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:47:52.383091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43913 (local address 127.0.98.3:47085)
0927 12:47:52.383695 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0927 12:47:52.383759 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 12:47:52.384817 (+ 1058us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:47:52.385410 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:47:52.385420 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:47:52.385787 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:47:52.386497 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:47:52.386514 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:47:52.387587 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:47:52.387591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:47:52.388993 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:47:52.389004 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:47:52.389736 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:47:52.389740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:47:52.389927 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:47:52.390639 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:47:52.390667 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:47:52.392445 (+ 1778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:47:52.394608 (+ 2163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:47:52.394615 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:47:52.394634 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:47:52.395004 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:47:52.395293 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:47:52.395296 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:47:52.395298 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:47:52.395502 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:47:52.395933 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:47:52.395941 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:47:52.396404 (+ 463us) client_negotiation.cc:769] Sending connection context
0927 12:47:52.396673 (+ 269us) client_negotiation.cc:240] Negotiation successful
0927 12:47:52.397047 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":187,"threads_started":1}
I20260927 12:47:52.400261 12112 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43913
I20260927 12:47:52.400522 12112 heartbeater.cc:464] Registering TS with master...
I20260927 12:47:52.401072 12112 heartbeater.cc:511] Master 127.0.98.62:43913 requested a full tablet report, sending...
I20260927 12:47:52.402242 11628 ts_manager.cc:194] Registered new tserver with Master: 1bbd576cc38544f68695229667c560e0 (127.0.98.3:45913)
I20260927 12:47:52.402956 11628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47085
I20260927 12:47:52.412420 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 12:47:52.425434 11845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:47:52.415502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35086 (local address 127.0.98.62:43913)
0927 12:47:52.415791 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0927 12:47:52.415796 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:47:52.415861 (+ 65us) server_negotiation.cc:407] Connection header received
0927 12:47:52.416008 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:47:52.416011 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:47:52.416063 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:47:52.416145 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:47:52.417065 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:47:52.417900 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:47:52.419100 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:47:52.419444 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:47:52.419550 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:47:52.421215 (+ 1665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:47:52.421219 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:47:52.421222 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:47:52.421250 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:47:52.423001 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:47:52.423472 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:47:52.423476 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:47:52.423478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:47:52.423569 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:47:52.423890 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:47:52.423893 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:47:52.423895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:47:52.424158 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:47:52.424295 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 12:47:52.425103 (+ 808us) server_negotiation.cc:299] Negotiation successful
0927 12:47:52.425258 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202}
I20260927 12:47:52.429231 11628 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35086:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:47:52.433758 11628 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:47:52.455765 12126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:47:52.445937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45913 (local address 127.0.0.1:38812)
0927 12:47:52.450153 (+ 4216us) negotiation.cc:107] Waiting for socket to connect
0927 12:47:52.450167 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:47:52.450270 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:47:52.450944 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:47:52.450947 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:47:52.450963 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:47:52.451046 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:47:52.451052 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:47:52.452550 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:47:52.452553 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:47:52.453862 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:47:52.453872 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:47:52.455187 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:47:52.455191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:47:52.455326 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:47:52.455371 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 12:47:52.455482 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 12:47:52.455606 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4155,"thread_start_us":114,"threads_started":1}
I20260927 12:47:52.456365 12128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:47:52.446188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50346 (local address 127.0.98.1:42893)
0927 12:47:52.447331 (+ 1143us) server_negotiation.cc:206] Beginning negotiation
0927 12:47:52.447338 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 12:47:52.449728 (+ 2390us) server_negotiation.cc:407] Connection header received
0927 12:47:52.449815 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:47:52.449821 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:47:52.450180 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:47:52.450337 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:47:52.450837 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:47:52.452072 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:47:52.454470 (+ 2398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:47:52.455312 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:47:52.455414 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:47:52.455561 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 12:47:52.456123 (+ 562us) server_negotiation.cc:299] Negotiation successful
0927 12:47:52.456230 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1034,"thread_start_us":69,"threads_started":1}
I20260927 12:47:52.456385 12129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:47:52.446369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38812 (local address 127.0.98.3:45913)
0927 12:47:52.447492 (+ 1123us) server_negotiation.cc:206] Beginning negotiation
0927 12:47:52.447503 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 12:47:52.450367 (+ 2864us) server_negotiation.cc:407] Connection header received
0927 12:47:52.450445 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:47:52.450450 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:47:52.450643 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:47:52.450798 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:47:52.451172 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:47:52.452420 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:47:52.454011 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:47:52.454865 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:47:52.454965 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:47:52.455121 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 12:47:52.456126 (+ 1005us) server_negotiation.cc:299] Negotiation successful
0927 12:47:52.456230 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1022,"thread_start_us":88,"threads_started":1}
I20260927 12:47:52.456022 12124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:47:52.445727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42893 (local address 127.0.0.1:50346)
0927 12:47:52.449497 (+ 3770us) negotiation.cc:107] Waiting for socket to connect
0927 12:47:52.449512 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:47:52.449611 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:47:52.450521 (+ 910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:47:52.450528 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:47:52.450551 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:47:52.450702 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:47:52.450710 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:47:52.452201 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:47:52.452204 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:47:52.454339 (+ 2135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:47:52.454350 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:47:52.455624 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:47:52.455628 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:47:52.455731 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:47:52.455754 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 12:47:52.455822 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 12:47:52.455883 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3702,"thread_start_us":95,"threads_started":1}
I20260927 12:47:52.460752 11776 tablet_service.cc:1467] Processing CreateTablet for tablet f2d33a7e3c054d3aa39321100687fa52 (DEFAULT_TABLE table=test-table [id=a84b3038d7cf44f7ae5ad9f9eec4bffb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:47:52.460754 12046 tablet_service.cc:1467] Processing CreateTablet for tablet f2d33a7e3c054d3aa39321100687fa52 (DEFAULT_TABLE table=test-table [id=a84b3038d7cf44f7ae5ad9f9eec4bffb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:47:52.461905 12046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2d33a7e3c054d3aa39321100687fa52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:47:52.461953 11776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2d33a7e3c054d3aa39321100687fa52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:47:52.471714 12131 tablet_bootstrap.cc:492] T f2d33a7e3c054d3aa39321100687fa52 P bb18193f93d6478191bffcf0d6700b09: Bootstrap starting.
I20260927 12:47:52.475302 12131 tablet_bootstrap.cc:654] T f2d33a7e3c054d3aa39321100687fa52 P bb18193f93d6478191bffcf0d6700b09: Neither blocks nor log segments found. Creating new log.
I20260927 12:47:52.475342 12127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:47:52.444519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41877 (local address 127.0.0.1:53586)
0927 12:47:52.447015 (+ 2496us) negotiation.cc:107] Waiting for socket to connect
0927 12:47:52.447071 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 12:47:52.447199 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:47:52.457292 (+ 10093us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:47:52.457295 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:47:52.457313 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:47:52.457400 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:47:52.457406 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:47:52.472349 (+ 14943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:47:52.472353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:47:52.473586 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:47:52.473598 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:47:52.474888 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:47:52.474892 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:47:52.474990 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:47:52.475023 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 12:47:52.475080 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 12:47:52.475170 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2313,"thread_start_us":104,"threads_started":1}
I20260927 12:47:52.476095 12125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:47:52.444677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53586 (local address 127.0.98.2:41877)
0927 12:47:52.456754 (+ 12077us) server_negotiation.cc:206] Beginning negotiation
0927 12:47:52.456769 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 12:47:52.456797 (+ 28us) server_negotiation.cc:407] Connection header received
0927 12:47:52.456879 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:47:52.456884 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:47:52.457115 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:47:52.457545 (+ 430us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:47:52.470967 (+ 13422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:47:52.472184 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:47:52.473772 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:47:52.474661 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:47:52.475625 (+ 964us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:47:52.475795 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 12:47:52.475850 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 12:47:52.475951 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11959,"thread_start_us":128,"threads_started":1}
I20260927 12:47:52.476475 12131 log.cc:824] T f2d33a7e3c054d3aa39321100687fa52 P bb18193f93d6478191bffcf0d6700b09: Log is configured to *not* fsync() on all Append() calls
I20260927 12:47:52.477906 12130 tablet_bootstrap.cc:492] T f2d33a7e3c054d3aa39321100687fa52 P 1bbd576cc38544f68695229667c560e0: Bootstrap starting.
I20260927 12:47:52.479429 11909 tablet_service.cc:1467] Processing CreateTablet for tablet f2d33a7e3c054d3aa39321100687fa52 (DEFAULT_TABLE table=test-table [id=a84b3038d7cf44f7ae5ad9f9eec4bffb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:47:52.480793 11909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2d33a7e3c054d3aa39321100687fa52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:47:52.481812 12130 tablet_bootstrap.cc:654] T f2d33a7e3c054d3aa39321100687fa52 P 1bbd576cc38544f68695229667c560e0: Neither blocks nor log segments found. Creating new log.
I20260927 12:47:52.482755 12130 log.cc:824] T f2d33a7e3c054d3aa39321100687fa52 P 1bbd576cc38544f68695229667c560e0: Log is configured to *not* fsync() on all Append() calls
I20260927 12:47:52.485105 12131 tablet_bootstrap.cc:492] T f2d33a7e3c054d3aa39321100687fa52 P bb18193f93d6478191bffcf0d6700b09: No bootstrap required, opened a new log
I20260927 12:47:52.485329 12131 ts_tablet_manager.cc:1397] T f2d33a7e3c054d3aa39321100687fa52 P bb18193f93d6478191bffcf0d6700b09: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20260927 12:47:52.489869 12134 tablet_bootstrap.cc:492] T f2d33a7e3c054d3aa39321100687fa52 P 515cff1da7a946da90528a90c91540d4: Bootstrap starting.
I20260927 12:47:52.491099 12131 raft_consensus.cc:348] T f2d33a7e3c054d3aa39321100687fa52 P bb18193f93d6478191bffcf0d6700b09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "515cff1da7a946da90528a90c91540d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41877 } } peers { permanent_uuid: "bb18193f93d6478191bffcf0d6700b09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42893 } } peers { permanent_uuid: "1bbd576cc38544f68695229667c560e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45913 } }
I20260927 12:47:52.491411 12131 raft_consensus.cc:374] T f2d33a7e3c054d3aa39321100687fa52 P bb18193f93d6478191bffcf0d6700b09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:47:52.491469 12131 raft_consensus.cc:729] T f2d33a7e3c054d3aa39321100687fa52 P bb18193f93d6478191bffcf0d6700b09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb18193f93d6478191bffcf0d6700b09, State: Initialized, Role: FOLLOWER
I20260927 12:47:52.492043 12131 consensus_queue.cc:260] T f2d33a7e3c054d3aa39321100687fa52 P bb18193f93d6478191bffcf0d6700b09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "515cff1da7a946da90528a90c91540d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41877 } } peers { permanent_uuid: "bb18193f93d6478191bffcf0d6700b09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42893 } } peers { permanent_uuid: "1bbd576cc38544f68695229667c560e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45913 } }
I20260927 12:47:52.492857 12134 tablet_bootstrap.cc:654] T f2d33a7e3c054d3aa39321100687fa52 P 515cff1da7a946da90528a90c91540d4: Neither blocks nor log segments found. Creating new log.
I20260927 12:47:52.493299 12131 ts_tablet_manager.cc:1428] T f2d33a7e3c054d3aa39321100687fa52 P bb18193f93d6478191bffcf0d6700b09: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260927 12:47:52.493714 11842 heartbeater.cc:503] Master 127.0.98.62:43913 was elected leader, sending a full tablet report...
I20260927 12:47:52.494402 12134 log.cc:824] T f2d33a7e3c054d3aa39321100687fa52 P 515cff1da7a946da90528a90c91540d4: Log is configured to *not* fsync() on all Append() calls
I20260927 12:47:52.495975 12130 tablet_bootstrap.cc:492] T f2d33a7e3c054d3aa39321100687fa52 P 1bbd576cc38544f68695229667c560e0: No bootstrap required, opened a new log
I20260927 12:47:52.496225 12130 ts_tablet_manager.cc:1397] T f2d33a7e3c054d3aa39321100687fa52 P 1bbd576cc38544f68695229667c560e0: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.000s
I20260927 12:47:52.496591 12134 tablet_bootstrap.cc:492] T f2d33a7e3c054d3aa39321100687fa52 P 515cff1da7a946da90528a90c91540d4: No bootstrap required, opened a new log
I20260927 12:47:52.496804 12134 ts_tablet_manager.cc:1397] T f2d33a7e3c054d3aa39321100687fa52 P 515cff1da7a946da90528a90c91540d4: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260927 12:47:52.500470 12130 raft_consensus.cc:348] T f2d33a7e3c054d3aa39321100687fa52 P 1bbd576cc38544f68695229667c560e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "515cff1da7a946da90528a90c91540d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41877 } } peers { permanent_uuid: "bb18193f93d6478191bffcf0d6700b09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42893 } } peers { permanent_uuid: "1bbd576cc38544f68695229667c560e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45913 } }
I20260927 12:47:52.501011 12134 raft_consensus.cc:348] T f2d33a7e3c054d3aa39321100687fa52 P 515cff1da7a946da90528a90c91540d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "515cff1da7a946da90528a90c91540d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41877 } } peers { permanent_uuid: "bb18193f93d6478191bffcf0d6700b09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42893 } } peers { permanent_uuid: "1bbd576cc38544f68695229667c560e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45913 } }
I20260927 12:47:52.501289 12134 raft_consensus.cc:374] T f2d33a7e3c054d3aa39321100687fa52 P 515cff1da7a946da90528a90c91540d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:47:52.501340 12134 raft_consensus.cc:729] T f2d33a7e3c054d3aa39321100687fa52 P 515cff1da7a946da90528a90c91540d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 515cff1da7a946da90528a90c91540d4, State: Initialized, Role: FOLLOWER
I20260927 12:47:52.501888 12134 consensus_queue.cc:260] T f2d33a7e3c054d3aa39321100687fa52 P 515cff1da7a946da90528a90c91540d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "515cff1da7a946da90528a90c91540d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41877 } } peers { permanent_uuid: "bb18193f93d6478191bffcf0d6700b09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42893 } } peers { permanent_uuid: "1bbd576cc38544f68695229667c560e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45913 } }
I20260927 12:47:52.503058 12134 ts_tablet_manager.cc:1428] T f2d33a7e3c054d3aa39321100687fa52 P 515cff1da7a946da90528a90c91540d4: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 12:47:52.503418 11978 heartbeater.cc:503] Master 127.0.98.62:43913 was elected leader, sending a full tablet report...
I20260927 12:47:52.503612 12130 raft_consensus.cc:374] T f2d33a7e3c054d3aa39321100687fa52 P 1bbd576cc38544f68695229667c560e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:47:52.503721 12130 raft_consensus.cc:729] T f2d33a7e3c054d3aa39321100687fa52 P 1bbd576cc38544f68695229667c560e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bbd576cc38544f68695229667c560e0, State: Initialized, Role: FOLLOWER
I20260927 12:47:52.504259 12130 consensus_queue.cc:260] T f2d33a7e3c054d3aa39321100687fa52 P 1bbd576cc38544f68695229667c560e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "515cff1da7a946da90528a90c91540d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41877 } } peers { permanent_uuid: "bb18193f93d6478191bffcf0d6700b09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42893 } } peers { permanent_uuid: "1bbd576cc38544f68695229667c560e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45913 } }
I20260927 12:47:52.505390 12130 ts_tablet_manager.cc:1428] T f2d33a7e3c054d3aa39321100687fa52 P 1bbd576cc38544f68695229667c560e0: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20260927 12:47:52.505780 12112 heartbeater.cc:503] Master 127.0.98.62:43913 was elected leader, sending a full tablet report...
Sep 27 12:47:52 dist-test-slave-lkn4 krb5kdc[11152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513271, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 12:47:52.621253 12137 raft_consensus.cc:482] T f2d33a7e3c054d3aa39321100687fa52 P 515cff1da7a946da90528a90c91540d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:47:52.621611 12137 raft_consensus.cc:504] T f2d33a7e3c054d3aa39321100687fa52 P 515cff1da7a946da90528a90c91540d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "515cff1da7a946da90528a90c91540d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41877 } } peers { permanent_uuid: "bb18193f93d6478191bffcf0d6700b09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42893 } } peers { permanent_uuid: "1bbd576cc38544f68695229667c560e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45913 } }
I20260927 12:47:52.623035 12137 leader_election.cc:288] T f2d33a7e3c054d3aa39321100687fa52 P 515cff1da7a946da90528a90c91540d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb18193f93d6478191bffcf0d6700b09 (127.0.98.1:42893), 1bbd576cc38544f68695229667c560e0 (127.0.98.3:45913)
I20260927 12:47:52.628911 12128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:47:52.623666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49249 (local address 127.0.98.1:42893)
0927 12:47:52.623800 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 12:47:52.623806 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:47:52.624024 (+ 218us) server_negotiation.cc:407] Connection header received
0927 12:47:52.624221 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:47:52.624226 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:47:52.624287 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:47:52.624347 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:47:52.624844 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:47:52.625618 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:47:52.627353 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:47:52.628023 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:47:52.628120 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:47:52.628219 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 12:47:52.628654 (+ 435us) server_negotiation.cc:299] Negotiation successful
0927 12:47:52.628750 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 12:47:52.629806 12140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:47:52.623244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42893 (local address 127.0.98.2:49249)
0927 12:47:52.623930 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0927 12:47:52.623946 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:47:52.624065 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:47:52.624520 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:47:52.624523 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:47:52.624544 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:47:52.624681 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:47:52.624689 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:47:52.625787 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:47:52.625791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:47:52.627158 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:47:52.627170 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:47:52.628375 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:47:52.628379 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:47:52.628492 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:47:52.628521 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 12:47:52.629521 (+ 1000us) client_negotiation.cc:240] Negotiation successful
0927 12:47:52.629624 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":76,"threads_started":1}
I20260927 12:47:52.630749 11792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2d33a7e3c054d3aa39321100687fa52" candidate_uuid: "515cff1da7a946da90528a90c91540d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb18193f93d6478191bffcf0d6700b09" is_pre_election: true
W20260927 12:47:52.630993 12113 tablet.cc:2369] T f2d33a7e3c054d3aa39321100687fa52 P 1bbd576cc38544f68695229667c560e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:47:52.631090 11792 raft_consensus.cc:2454] T f2d33a7e3c054d3aa39321100687fa52 P bb18193f93d6478191bffcf0d6700b09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 515cff1da7a946da90528a90c91540d4 in term 0.
I20260927 12:47:52.632126 11865 leader_election.cc:302] T f2d33a7e3c054d3aa39321100687fa52 P 515cff1da7a946da90528a90c91540d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 515cff1da7a946da90528a90c91540d4, bb18193f93d6478191bffcf0d6700b09; no voters:
I20260927 12:47:52.632642 12137 raft_consensus.cc:2781] T f2d33a7e3c054d3aa39321100687fa52 P 515cff1da7a946da90528a90c91540d4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:47:52.632746 12137 raft_consensus.cc:482] T f2d33a7e3c054d3aa39321100687fa52 P 515cff1da7a946da90528a90c91540d4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:47:52.632805 12137 raft_consensus.cc:3037] T f2d33a7e3c054d3aa39321100687fa52 P 515cff1da7a946da90528a90c91540d4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:47:52.634157 12137 raft_consensus.cc:504] T f2d33a7e3c054d3aa39321100687fa52 P 515cff1da7a946da90528a90c91540d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "515cff1da7a946da90528a90c91540d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41877 } } peers { permanent_uuid: "bb18193f93d6478191bffcf0d6700b09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42893 } } peers { permanent_uuid: "1bbd576cc38544f68695229667c560e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45913 } }
I20260927 12:47:52.634539 12137 leader_election.cc:288] T f2d33a7e3c054d3aa39321100687fa52 P 515cff1da7a946da90528a90c91540d4 [CANDIDATE]: Term 1 election: Requested vote from peers bb18193f93d6478191bffcf0d6700b09 (127.0.98.1:42893), 1bbd576cc38544f68695229667c560e0 (127.0.98.3:45913)
I20260927 12:47:52.635113 11792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2d33a7e3c054d3aa39321100687fa52" candidate_uuid: "515cff1da7a946da90528a90c91540d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb18193f93d6478191bffcf0d6700b09"
I20260927 12:47:52.635255 11792 raft_consensus.cc:3037] T f2d33a7e3c054d3aa39321100687fa52 P bb18193f93d6478191bffcf0d6700b09 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:47:52.636504 11792 raft_consensus.cc:2454] T f2d33a7e3c054d3aa39321100687fa52 P bb18193f93d6478191bffcf0d6700b09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 515cff1da7a946da90528a90c91540d4 in term 1.
I20260927 12:47:52.636898 11865 leader_election.cc:302] T f2d33a7e3c054d3aa39321100687fa52 P 515cff1da7a946da90528a90c91540d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 515cff1da7a946da90528a90c91540d4, bb18193f93d6478191bffcf0d6700b09; no voters:
I20260927 12:47:52.637146 12137 raft_consensus.cc:2781] T f2d33a7e3c054d3aa39321100687fa52 P 515cff1da7a946da90528a90c91540d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:47:52.637382 12137 raft_consensus.cc:686] T f2d33a7e3c054d3aa39321100687fa52 P 515cff1da7a946da90528a90c91540d4 [term 1 LEADER]: Becoming Leader. State: Replica: 515cff1da7a946da90528a90c91540d4, State: Running, Role: LEADER
I20260927 12:47:52.637766 12137 consensus_queue.cc:237] T f2d33a7e3c054d3aa39321100687fa52 P 515cff1da7a946da90528a90c91540d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "515cff1da7a946da90528a90c91540d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41877 } } peers { permanent_uuid: "bb18193f93d6478191bffcf0d6700b09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42893 } } peers { permanent_uuid: "1bbd576cc38544f68695229667c560e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45913 } }
I20260927 12:47:52.638595 12141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:47:52.623326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45913 (local address 127.0.98.2:53593)
0927 12:47:52.624296 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0927 12:47:52.624311 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:47:52.624724 (+ 413us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:47:52.625100 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:47:52.625103 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:47:52.625120 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:47:52.625252 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:47:52.625258 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:47:52.626274 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:47:52.626277 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:47:52.627447 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:47:52.627457 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:47:52.638118 (+ 10661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:47:52.638123 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:47:52.638244 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:47:52.638275 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 12:47:52.638364 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 12:47:52.638450 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":888,"thread_start_us":110,"threads_started":1}
I20260927 12:47:52.638659 12129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:47:52.623687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53593 (local address 127.0.98.3:45913)
0927 12:47:52.623899 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 12:47:52.623904 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:47:52.624386 (+ 482us) server_negotiation.cc:407] Connection header received
0927 12:47:52.624873 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:47:52.624877 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:47:52.624926 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:47:52.624970 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:47:52.625379 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:47:52.626105 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:47:52.627566 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:47:52.628088 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:47:52.628186 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:47:52.628250 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 12:47:52.638424 (+ 10174us) server_negotiation.cc:299] Negotiation successful
0927 12:47:52.638538 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
I20260927 12:47:52.639477 12066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2d33a7e3c054d3aa39321100687fa52" candidate_uuid: "515cff1da7a946da90528a90c91540d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bbd576cc38544f68695229667c560e0" is_pre_election: true
I20260927 12:47:52.639487 12065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2d33a7e3c054d3aa39321100687fa52" candidate_uuid: "515cff1da7a946da90528a90c91540d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bbd576cc38544f68695229667c560e0"
I20260927 12:47:52.639716 12065 raft_consensus.cc:3037] T f2d33a7e3c054d3aa39321100687fa52 P 1bbd576cc38544f68695229667c560e0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:47:52.641026 12065 raft_consensus.cc:2454] T f2d33a7e3c054d3aa39321100687fa52 P 1bbd576cc38544f68695229667c560e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 515cff1da7a946da90528a90c91540d4 in term 1.
I20260927 12:47:52.641311 11627 catalog_manager.cc:5495] T f2d33a7e3c054d3aa39321100687fa52 P 515cff1da7a946da90528a90c91540d4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 515cff1da7a946da90528a90c91540d4 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "515cff1da7a946da90528a90c91540d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "515cff1da7a946da90528a90c91540d4" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb18193f93d6478191bffcf0d6700b09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bbd576cc38544f68695229667c560e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45913 } health_report { overall_health: UNKNOWN } } }
W20260927 12:47:52.673259 11843 tablet.cc:2369] T f2d33a7e3c054d3aa39321100687fa52 P bb18193f93d6478191bffcf0d6700b09: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:47:52.704198 12125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:47:52.698696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53602 (local address 127.0.98.2:41877)
0927 12:47:52.698999 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0927 12:47:52.699003 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:47:52.699028 (+ 25us) server_negotiation.cc:407] Connection header received
0927 12:47:52.699121 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:47:52.699123 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:47:52.699187 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:47:52.699299 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:47:52.699657 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:47:52.700432 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:47:52.701731 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:47:52.702092 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:47:52.702188 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:47:52.702673 (+ 485us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:47:52.703320 (+ 647us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:47:52.703440 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 12:47:52.703969 (+ 529us) server_negotiation.cc:299] Negotiation successful
0927 12:47:52.704035 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219}
I20260927 12:47:52.720906 12065 raft_consensus.cc:1261] T f2d33a7e3c054d3aa39321100687fa52 P 1bbd576cc38544f68695229667c560e0 [term 1 FOLLOWER]: Refusing update from remote peer 515cff1da7a946da90528a90c91540d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:47:52.721838 12137 consensus_queue.cc:1035] T f2d33a7e3c054d3aa39321100687fa52 P 515cff1da7a946da90528a90c91540d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bbd576cc38544f68695229667c560e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:47:52.729348 11792 raft_consensus.cc:1261] T f2d33a7e3c054d3aa39321100687fa52 P bb18193f93d6478191bffcf0d6700b09 [term 1 FOLLOWER]: Refusing update from remote peer 515cff1da7a946da90528a90c91540d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:47:52.730135 12137 consensus_queue.cc:1035] T f2d33a7e3c054d3aa39321100687fa52 P 515cff1da7a946da90528a90c91540d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb18193f93d6478191bffcf0d6700b09" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 12:47:52.730484 11979 tablet.cc:2369] T f2d33a7e3c054d3aa39321100687fa52 P 515cff1da7a946da90528a90c91540d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:47:52.735309 12147 mvcc.cc:205] Tried to move back new op lower bound from 7333942365056352256 to 7333942364727947264. Current Snapshot: MvccSnapshot[applied={T|T < 7333942365056352256}]
I20260927 12:47:52.735455 12146 mvcc.cc:205] Tried to move back new op lower bound from 7333942365056352256 to 7333942364727947264. Current Snapshot: MvccSnapshot[applied={T|T < 7333942365056352256}]
I20260927 12:47:52.745213 11627 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:47:52.745404 11627 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:47:52.749068 11627 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0a02703ac1334a629da126a949285530: Sending DeleteTablet for 3 replicas of tablet f2d33a7e3c054d3aa39321100687fa52
I20260927 12:47:52.750861 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 11709
I20260927 12:47:52.751430 11909 tablet_service.cc:1514] Processing DeleteTablet for tablet f2d33a7e3c054d3aa39321100687fa52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:47:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53586
I20260927 12:47:52.752028 12046 tablet_service.cc:1514] Processing DeleteTablet for tablet f2d33a7e3c054d3aa39321100687fa52 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:47:52 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38812
I20260927 12:47:52.754838 12154 tablet_replica.cc:330] T f2d33a7e3c054d3aa39321100687fa52 P 515cff1da7a946da90528a90c91540d4: stopping tablet replica
I20260927 12:47:52.755200 12154 raft_consensus.cc:2229] T f2d33a7e3c054d3aa39321100687fa52 P 515cff1da7a946da90528a90c91540d4 [term 1 LEADER]: Raft consensus shutting down.
I20260927 12:47:52.755674 12154 raft_consensus.cc:2258] T f2d33a7e3c054d3aa39321100687fa52 P 515cff1da7a946da90528a90c91540d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:47:52.757874 12154 ts_tablet_manager.cc:1905] T f2d33a7e3c054d3aa39321100687fa52 P 515cff1da7a946da90528a90c91540d4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 12:47:52.760526 11865 connection.cc:535] client connection to 127.0.98.1:42893 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260927 12:47:52.760689 11865 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 12:47:52.761427 12155 tablet_replica.cc:330] T f2d33a7e3c054d3aa39321100687fa52 P 1bbd576cc38544f68695229667c560e0: stopping tablet replica
I20260927 12:47:52.761895 12155 raft_consensus.cc:2229] T f2d33a7e3c054d3aa39321100687fa52 P 1bbd576cc38544f68695229667c560e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:47:52.767967 12154 ts_tablet_manager.cc:1918] T f2d33a7e3c054d3aa39321100687fa52 P 515cff1da7a946da90528a90c91540d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:47:52.770504 12154 log.cc:1196] T f2d33a7e3c054d3aa39321100687fa52 P 515cff1da7a946da90528a90c91540d4: Deleting WAL directory at /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/ts-1/wal/wals/f2d33a7e3c054d3aa39321100687fa52
I20260927 12:47:52.770537 12155 raft_consensus.cc:2258] T f2d33a7e3c054d3aa39321100687fa52 P 1bbd576cc38544f68695229667c560e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:47:52.771139 12154 ts_tablet_manager.cc:1939] T f2d33a7e3c054d3aa39321100687fa52 P 515cff1da7a946da90528a90c91540d4: Deleting consensus metadata
I20260927 12:47:52.772658 11615 catalog_manager.cc:4841] TS 515cff1da7a946da90528a90c91540d4 (127.0.98.2:41877): tablet f2d33a7e3c054d3aa39321100687fa52 (table test-table [id=a84b3038d7cf44f7ae5ad9f9eec4bffb]) successfully deleted
W20260927 12:47:52.773062 11612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42893 (error 108)
W20260927 12:47:52.773952 12141 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:47:52.772921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42893 (local address 127.0.98.2:52999)
0927 12:47:52.773113 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 12:47:52.773128 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:47:52.773216 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:47:52.773863 (+ 647us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42893: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":117}
I20260927 12:47:52.774345 12155 ts_tablet_manager.cc:1905] T f2d33a7e3c054d3aa39321100687fa52 P 1bbd576cc38544f68695229667c560e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:47:52.774605 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 11848
I20260927 12:47:52.779337 12155 ts_tablet_manager.cc:1918] T f2d33a7e3c054d3aa39321100687fa52 P 1bbd576cc38544f68695229667c560e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:47:52.779513 12155 log.cc:1196] T f2d33a7e3c054d3aa39321100687fa52 P 1bbd576cc38544f68695229667c560e0: Deleting WAL directory at /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790513161160093-392-0/minicluster-data/ts-2/wal/wals/f2d33a7e3c054d3aa39321100687fa52
I20260927 12:47:52.779919 12155 ts_tablet_manager.cc:1939] T f2d33a7e3c054d3aa39321100687fa52 P 1bbd576cc38544f68695229667c560e0: Deleting consensus metadata
I20260927 12:47:52.781423 11613 catalog_manager.cc:4841] TS 1bbd576cc38544f68695229667c560e0 (127.0.98.3:45913): tablet f2d33a7e3c054d3aa39321100687fa52 (table test-table [id=a84b3038d7cf44f7ae5ad9f9eec4bffb]) successfully deleted
I20260927 12:47:52.782264 12127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:47:52.781906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42893 (local address 127.0.0.1:50356)
0927 12:47:52.782104 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 12:47:52.782179 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
W20260927 12:47:52.782462 11612 catalog_manager.cc:4584] TS bb18193f93d6478191bffcf0d6700b09 (127.0.98.1:42893): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f2d33a7e3c054d3aa39321100687fa52: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42893: connect: Connection refused (error 111)
I20260927 12:47:52.787792 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 11983
I20260927 12:47:52.795135 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 11596
I20260927 12:47:52.800676 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 12:47:53.135540 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 12:47:53.135694 392 mini_ranger.cc:66] Stopping Ranger...
I20260927 12:47:53.470515 392 mini_ranger.cc:68] Stopped Ranger
2026-09-27 12:47:53.470 UTC [11182] LOG: received smart shutdown request
2026-09-27 12:47:53.474 UTC [11182] LOG: background worker "logical replication launcher" (PID 11191) exited with exit code 1
2026-09-27 12:47:53.474 UTC [11186] LOG: shutting down
2026-09-27 12:47:53.650 UTC [11182] LOG: database system is shut down
2026-09-27T12:47:53Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46570 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:47:53 dist-test-slave-lkn4 krb5kdc[12166](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:47:53 dist-test-slave-lkn4 krb5kdc[12166](info): set up 2 sockets
Sep 27 12:47:53 dist-test-slave-lkn4 krb5kdc[12166](info): commencing operation
krb5kdc: starting...
W20260927 12:47:54.701936 392 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/krb5kdc/test-user.keytab.
Sep 27 12:47:54 dist-test-slave-lkn4 krb5kdc[12166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513274, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:47:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:47:54Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260927 12:47:54.870642 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskTC8Et9/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/postgres -l logfile start
2026-09-27 12:47:57.449 UTC [12196] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 12:47:57.449 UTC [12196] LOG: listening on IPv4 address "127.0.98.20", port 46929
2026-09-27 12:47:57.453 UTC [12196] LOG: listening on Unix socket "/tmp/.s.PGSQL.46929"
2026-09-27 12:47:57.481 UTC [12199] LOG: database system was shut down at 2026-09-27 12:47:55 UTC
2026-09-27 12:47:57.485 UTC [12196] LOG: database system is ready to accept connections
I20260927 12:47:58.350097 392 mini_postgres.cc:96] Postgres bound to 46929
2026-09-27 12:47:58.355 UTC [12207] FATAL: database "slave" does not exist
127.0.98.20:46929 - accepting connections
I20260927 12:47:58.356504 392 mini_ranger.cc:161] Starting Ranger...
I20260927 12:47:58.370970 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 12:47:58.877964 392 mini_ranger.cc:87] Created ranger Postgres database
I20260927 12:47:58.878069 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ranger-admin
2026-09-27 12:47:59,160 [I] DB FLAVOR :POSTGRES
2026-09-27 12:47:59,160 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 12:47:59,160 [I] Checking connection..
2026-09-27 12:47:59,160 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTC8Et9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTC8Et9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 12:47:59,405 [I] Checking connection passed.
2026-09-27 12:47:59,405 [I] --------- Verifying version history table ---------
2026-09-27 12:47:59,406 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTC8Et9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTC8Et9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 12:47:59,632 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 12:47:59,632 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTC8Et9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTC8Et9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 12:47:59,865 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 12:47:59,865 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 12:47:59,865 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTC8Et9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTC8Et9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46929/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskTC8Et9/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 12:48:00.094 UTC [12271] WARNING: there is no transaction in progress
2026-09-27 12:48:00,099 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 12:48:00,100 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTC8Et9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTC8Et9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 12:48:00,340 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 12:48:00,341 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 12:48:00,341 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTC8Et9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTC8Et9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 12:48:00,578 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTC8Et9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTC8Et9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 12:48:00,819 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTC8Et9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTC8Et9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-lkn4.c.gcp-upstream.internal','N') ;"
2026-09-27 12:48:01,047 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTC8Et9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTC8Et9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 12:48:01,279 [I] Table x_portal_user does not exist in database ranger
2026-09-27 12:48:01,279 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTC8Et9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTC8Et9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 12:48:01,544 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 12:48:01,544 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTC8Et9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTC8Et9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 12:48:01,779 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTC8Et9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTC8Et9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 12:48:02,015 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 12:48:02,015 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTC8Et9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTC8Et9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46929/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskTC8Et9/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 12:48:03.010 UTC [12379] WARNING: there is no transaction in progress
2026-09-27 12:48:03.026 UTC [12379] WARNING: there is no transaction in progress
2026-09-27 12:48:03.042 UTC [12379] WARNING: there is no transaction in progress
2026-09-27 12:48:03.058 UTC [12379] WARNING: there is no transaction in progress
2026-09-27 12:48:03.076 UTC [12379] WARNING: there is no transaction in progress
2026-09-27 12:48:03.097 UTC [12379] WARNING: there is no transaction in progress
2026-09-27 12:48:03.152 UTC [12379] WARNING: there is no transaction in progress
2026-09-27 12:48:03.158 UTC [12379] WARNING: there is no transaction in progress
2026-09-27 12:48:03.172 UTC [12379] WARNING: there is no transaction in progress
2026-09-27 12:48:03.182 UTC [12379] WARNING: there is no transaction in progress
2026-09-27 12:48:03.192 UTC [12379] WARNING: there is no transaction in progress
2026-09-27 12:48:03.205 UTC [12379] WARNING: there is no transaction in progress
2026-09-27 12:48:03.215 UTC [12379] WARNING: there is no transaction in progress
2026-09-27 12:48:03.222 UTC [12379] WARNING: there is no transaction in progress
2026-09-27 12:48:03.229 UTC [12379] WARNING: there is no transaction in progress
2026-09-27 12:48:03.235 UTC [12379] WARNING: there is no transaction in progress
2026-09-27 12:48:03,867 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 12:48:03,867 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTC8Et9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTC8Et9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 12:48:04,097 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 12:48:04,098 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTC8Et9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTC8Et9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 12:48:04,327 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 12:48:04,327 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTC8Et9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTC8Et9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 12:48:04,562 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 12:48:04,563 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTC8Et9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTC8Et9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 12:48:04,802 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTC8Et9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTC8Et9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 12:48:05,041 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTC8Et9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTC8Et9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-lkn4.c.gcp-upstream.internal';"
2026-09-27 12:48:05,276 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 12:48:05,276 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTC8Et9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTC8Et9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46929/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 12:48:05,518 [I] DB_PATCHES have already been applied
I20260927 12:48:05.523665 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskTC8Et9/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskTC8Et9/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskTC8Et9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTC8Et9/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 12:48:05.523736 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260927 12:48:05.526971 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:39389
Sep 27, 2026 12:48:05 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 12:48:05 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 12:48:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 12:48:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 12:48:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 12:48:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 12:48:06 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 12:48:06.524042 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.168s user 0.000s sys 0.009s
Sep 27, 2026 12:48:26 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 12:48:26 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 12:48:26 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 12:48:26 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 12:48:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 12:48:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 12:48:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 12:48:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 12:48:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 12:48:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 12:48:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 12:48:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 12:48:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 12:48:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 12:48:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 12:48:26 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 12:48:26 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 12:48:27 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 12:48:27.874812 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 12:48:27 dist-test-slave-lkn4 krb5kdc[12166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513307, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 12:48:27.960850 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 12:48:27.970716 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 12:48:28.942584 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 12:48:28.942788 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ranger-kms
I20260927 12:48:28.942811 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:46929
2026-09-27 12:48:29,096 [I] DB FLAVOR :POSTGRES
2026-09-27 12:48:29,097 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 12:48:29,097 [I] Checking connection
2026-09-27 12:48:29,097 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTC8Et9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46929/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 12:48:29,351 [I] connection success
2026-09-27 12:48:29,351 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 12:48:29,351 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 12:48:29,351 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTC8Et9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46929/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 12:48:29,585 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 12:48:29,585 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 12:48:29,585 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 12:48:29,585 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTC8Et9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46929/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 12:48:29,833 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 12:48:30.304431 392 mini_ranger_kms.cc:310] Created kms service
I20260927 12:48:30.521289 392 mini_ranger_kms.cc:326] Created kudu user
I20260927 12:48:30.570684 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 12:48:30 dist-test-slave-lkn4 krb5kdc[12166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513307, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 12:48:30.980767 392 mini_ranger_kms.cc:384] Added ranger policy
I20260927 12:48:30.981024 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskTC8Et9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskTC8Et9/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskTC8Et9/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 12:48:30.981055 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260927 12:48:30.984797 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12559
I20260927 12:48:30.984887 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:55541
Sep 27, 2026 12:48:31 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 12:48:31 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 12:48:32 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 12:48:32 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 12:48:32 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 12:48:32 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 12:48:32.983881 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.023s user 0.000s sys 0.138s
I20260927 12:48:32.983991 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 12:48:32.984066 392 mini_ranger_kms.cc:401] 127.0.98.20:55541/kms/v1/keys
Sep 27 12:48:33 dist-test-slave-lkn4 krb5kdc[12166](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790513313, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 12:48:33 dist-test-slave-lkn4 krb5kdc[12166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513313, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 12:48:33 dist-test-slave-lkn4 krb5kdc[12166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513313, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 12:48:36 dist-test-slave-lkn4 krb5kdc[12166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:48:36.185925 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32875
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32875
--ranger_config_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:55541/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:48:36.308544 12610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:48:36.308844 12610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:48:36.308883 12610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:48:36.312994 12610 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:48:36.313068 12610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:48:36.313086 12610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:48:36.313102 12610 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:48:36.313115 12610 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:48:36.318842 12610 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:55541/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32875
--ranger_config_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:48:36.319954 12610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:48:36.321013 12610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:48:36.330787 12616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:48:36.330796 12615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:48:36.330972 12618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:48:36.331076 12610 server_base.cc:1034] running on GCE node
I20260927 12:48:36.331764 12610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:48:36.333177 12610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:48:36.334407 12610 hybrid_clock.cc:648] HybridClock initialized: now 1790513316334374 us; error 53 us; skew 500 ppm
Sep 27 12:48:36 dist-test-slave-lkn4 krb5kdc[12166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:48:36.341848 12610 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:48:36.343145 12610 webserver.cc:460] Webserver started at http://127.0.98.62:36461/ using document root <none> and password file <none>
I20260927 12:48:36.343879 12610 fs_manager.cc:359] Metadata directory not provided
I20260927 12:48:36.343962 12610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:48:36.344185 12610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 12:48:36 dist-test-slave-lkn4 krb5kdc[12166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 12:48:36.409708 12610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/master-0/data/instance:
uuid: "125e81691bcc4d5f860760aa8a45362c"
format_stamp: "Formatted at 2026-09-27 12:48:36 on dist-test-slave-lkn4"
server_key: "8d0b6aa75498b459f26bc5431b85351b"
server_key_iv: "54a2ce4beaa4c7e257f0cade0ee912cb"
server_key_version: "kuduclusterkey@0"
I20260927 12:48:36.410408 12610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/master-0/wal/instance:
uuid: "125e81691bcc4d5f860760aa8a45362c"
format_stamp: "Formatted at 2026-09-27 12:48:36 on dist-test-slave-lkn4"
server_key: "8d0b6aa75498b459f26bc5431b85351b"
server_key_iv: "54a2ce4beaa4c7e257f0cade0ee912cb"
server_key_version: "kuduclusterkey@0"
I20260927 12:48:36.414412 12610 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 12:48:36.433868 12625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:48:36.435531 12610 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:48:36.435684 12610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/master-0/wal
uuid: "125e81691bcc4d5f860760aa8a45362c"
format_stamp: "Formatted at 2026-09-27 12:48:36 on dist-test-slave-lkn4"
server_key: "8d0b6aa75498b459f26bc5431b85351b"
server_key_iv: "54a2ce4beaa4c7e257f0cade0ee912cb"
server_key_version: "kuduclusterkey@0"
I20260927 12:48:36.435840 12610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:48:36.463596 12610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:48:36.467326 12610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:48:36.467527 12610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:48:36.476464 12610 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32875
I20260927 12:48:36.476465 12677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32875 every 8 connection(s)
I20260927 12:48:36.477869 12610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:48:36.480886 12678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:48:36.484150 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 12610
I20260927 12:48:36.484272 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/master-0/wal/instance
I20260927 12:48:36.484565 392 external_mini_cluster.cc:1428] Setting key a721408d7eb29e73d841ef6931af1f31
I20260927 12:48:36.487635 12678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 125e81691bcc4d5f860760aa8a45362c: Bootstrap starting.
I20260927 12:48:36.490139 12678 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 125e81691bcc4d5f860760aa8a45362c: Neither blocks nor log segments found. Creating new log.
I20260927 12:48:36.490938 12678 log.cc:824] T 00000000000000000000000000000000 P 125e81691bcc4d5f860760aa8a45362c: Log is configured to *not* fsync() on all Append() calls
I20260927 12:48:36.493142 12678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 125e81691bcc4d5f860760aa8a45362c: No bootstrap required, opened a new log
Sep 27 12:48:36 dist-test-slave-lkn4 krb5kdc[12166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:48:36.497116 12678 raft_consensus.cc:348] T 00000000000000000000000000000000 P 125e81691bcc4d5f860760aa8a45362c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "125e81691bcc4d5f860760aa8a45362c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32875 } }
I20260927 12:48:36.497354 12678 raft_consensus.cc:374] T 00000000000000000000000000000000 P 125e81691bcc4d5f860760aa8a45362c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:48:36.497432 12678 raft_consensus.cc:729] T 00000000000000000000000000000000 P 125e81691bcc4d5f860760aa8a45362c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125e81691bcc4d5f860760aa8a45362c, State: Initialized, Role: FOLLOWER
I20260927 12:48:36.497951 12678 consensus_queue.cc:260] T 00000000000000000000000000000000 P 125e81691bcc4d5f860760aa8a45362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "125e81691bcc4d5f860760aa8a45362c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32875 } }
I20260927 12:48:36.498111 12678 raft_consensus.cc:388] T 00000000000000000000000000000000 P 125e81691bcc4d5f860760aa8a45362c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:48:36.498175 12678 raft_consensus.cc:482] T 00000000000000000000000000000000 P 125e81691bcc4d5f860760aa8a45362c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:48:36.498303 12678 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 125e81691bcc4d5f860760aa8a45362c [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:48:36.499281 12678 raft_consensus.cc:504] T 00000000000000000000000000000000 P 125e81691bcc4d5f860760aa8a45362c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "125e81691bcc4d5f860760aa8a45362c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32875 } }
I20260927 12:48:36.499697 12678 leader_election.cc:302] T 00000000000000000000000000000000 P 125e81691bcc4d5f860760aa8a45362c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 125e81691bcc4d5f860760aa8a45362c; no voters:
I20260927 12:48:36.500011 12678 leader_election.cc:288] T 00000000000000000000000000000000 P 125e81691bcc4d5f860760aa8a45362c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:48:36.500131 12683 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 125e81691bcc4d5f860760aa8a45362c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:48:36.500329 12683 raft_consensus.cc:686] T 00000000000000000000000000000000 P 125e81691bcc4d5f860760aa8a45362c [term 1 LEADER]: Becoming Leader. State: Replica: 125e81691bcc4d5f860760aa8a45362c, State: Running, Role: LEADER
I20260927 12:48:36.500667 12681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:36.486017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59466 (local address 127.0.98.62:32875)
0927 12:48:36.486467 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0927 12:48:36.486485 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 12:48:36.486532 (+ 47us) server_negotiation.cc:407] Connection header received
0927 12:48:36.487366 (+ 834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:48:36.487389 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:48:36.487829 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:48:36.488191 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:48:36.489019 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:36.490567 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:36.491639 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:36.492277 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:36.492458 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:36.495443 (+ 2985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:48:36.495449 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:48:36.495466 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:48:36.495496 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:48:36.497402 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:48:36.497901 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:48:36.497906 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:48:36.497913 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:48:36.498064 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:48:36.498432 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:48:36.498435 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:48:36.498437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:48:36.498926 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:48:36.499096 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 12:48:36.499462 (+ 366us) server_negotiation.cc:299] Negotiation successful
0927 12:48:36.499740 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":102,"threads_started":1}
I20260927 12:48:36.501349 12678 sys_catalog.cc:564] T 00000000000000000000000000000000 P 125e81691bcc4d5f860760aa8a45362c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:48:36.500664 12683 consensus_queue.cc:237] T 00000000000000000000000000000000 P 125e81691bcc4d5f860760aa8a45362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "125e81691bcc4d5f860760aa8a45362c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32875 } }
I20260927 12:48:36.504076 12685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 125e81691bcc4d5f860760aa8a45362c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 125e81691bcc4d5f860760aa8a45362c. Latest consensus state: current_term: 1 leader_uuid: "125e81691bcc4d5f860760aa8a45362c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "125e81691bcc4d5f860760aa8a45362c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32875 } } }
I20260927 12:48:36.504132 12678 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 12:48:36.504278 12684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 125e81691bcc4d5f860760aa8a45362c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "125e81691bcc4d5f860760aa8a45362c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "125e81691bcc4d5f860760aa8a45362c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32875 } } }
I20260927 12:48:36.504366 12684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 125e81691bcc4d5f860760aa8a45362c [sys.catalog]: This master's current role is: LEADER
I20260927 12:48:36.505256 12685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 125e81691bcc4d5f860760aa8a45362c [sys.catalog]: This master's current role is: LEADER
I20260927 12:48:36.507741 12697 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:48:36.511759 12697 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:48:36.521903 12697 catalog_manager.cc:1347] Generated new cluster ID: 21710f5edef2472cb74025cc1b64cd7c
I20260927 12:48:36.521971 12697 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:48:36.540212 12697 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:48:36.541625 12697 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:48:36.550619 12697 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 125e81691bcc4d5f860760aa8a45362c: Generated new TSK 0
I20260927 12:48:36.551571 12697 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 12:48:37.272358 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:55541/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32875
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:48:37.437260 12723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:48:37.437747 12723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:48:37.437826 12723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:48:37.444114 12723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:48:37.444232 12723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:48:37.444331 12723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:48:37.452550 12723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:55541/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:48:37.454118 12723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:48:37.455572 12723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:48:37.468582 12733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:48:37.468698 12730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:48:37.468997 12731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:48:37.469376 12723 server_base.cc:1034] running on GCE node
I20260927 12:48:37.469887 12723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:48:37.470741 12723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:48:37.471939 12723 hybrid_clock.cc:648] HybridClock initialized: now 1790513317471905 us; error 43 us; skew 500 ppm
Sep 27 12:48:37 dist-test-slave-lkn4 krb5kdc[12166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513317, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:48:37.475481 12723 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 12:48:37.476673 12723 webserver.cc:460] Webserver started at http://127.0.98.1:44437/ using document root <none> and password file <none>
I20260927 12:48:37.477394 12723 fs_manager.cc:359] Metadata directory not provided
I20260927 12:48:37.477471 12723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:48:37.477691 12723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 12:48:37 dist-test-slave-lkn4 krb5kdc[12166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513317, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 12:48:37.501560 12723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-0/data/instance:
uuid: "c3de391db8bc406383d179ec7ab6e5f5"
format_stamp: "Formatted at 2026-09-27 12:48:37 on dist-test-slave-lkn4"
server_key: "845d5ab1f76e9a47dd25b60fe5fe78cb"
server_key_iv: "e372c974abac3612311e0226b8e4faad"
server_key_version: "kuduclusterkey@0"
I20260927 12:48:37.502349 12723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c3de391db8bc406383d179ec7ab6e5f5"
format_stamp: "Formatted at 2026-09-27 12:48:37 on dist-test-slave-lkn4"
server_key: "845d5ab1f76e9a47dd25b60fe5fe78cb"
server_key_iv: "e372c974abac3612311e0226b8e4faad"
server_key_version: "kuduclusterkey@0"
I20260927 12:48:37.507417 12723 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260927 12:48:37.527614 12740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:48:37.529356 12723 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 12:48:37.529512 12723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-0/wal
uuid: "c3de391db8bc406383d179ec7ab6e5f5"
format_stamp: "Formatted at 2026-09-27 12:48:37 on dist-test-slave-lkn4"
server_key: "845d5ab1f76e9a47dd25b60fe5fe78cb"
server_key_iv: "e372c974abac3612311e0226b8e4faad"
server_key_version: "kuduclusterkey@0"
I20260927 12:48:37.529664 12723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:48:37.559216 12723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:48:37.562810 12723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:48:37.563030 12723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:48:37.563711 12723 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:48:37.564759 12723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:48:37.564870 12723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:48:37.564942 12723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:48:37.564976 12723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:48:37.575218 12855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:32911 every 8 connection(s)
I20260927 12:48:37.575202 12723 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:32911
I20260927 12:48:37.576411 12723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:48:37.580921 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 12723
I20260927 12:48:37.581053 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:48:37.581351 392 external_mini_cluster.cc:1428] Setting key ae77709bdd44b06df70f9c25cfd452e1
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Sep 27 12:48:37 dist-test-slave-lkn4 krb5kdc[12166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513317, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 12:48:37.619145 12858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:37.578556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50927 (local address 127.0.98.62:32875)
0927 12:48:37.586283 (+ 7727us) server_negotiation.cc:206] Beginning negotiation
0927 12:48:37.586289 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:48:37.588807 (+ 2518us) server_negotiation.cc:407] Connection header received
0927 12:48:37.588866 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:48:37.588869 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:48:37.588935 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:48:37.589030 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:48:37.596429 (+ 7399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:37.597347 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:37.602191 (+ 4844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:37.602591 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:37.602688 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:37.614092 (+ 11404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:48:37.614097 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:48:37.614101 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:48:37.614140 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:48:37.616178 (+ 2038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:48:37.617027 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:48:37.617032 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:48:37.617034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:48:37.617135 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:48:37.617608 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:48:37.617611 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:48:37.617613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:48:37.617877 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:48:37.617967 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 12:48:37.618810 (+ 843us) server_negotiation.cc:299] Negotiation successful
0927 12:48:37.618963 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7617,"thread_start_us":123,"threads_started":1}
I20260927 12:48:37.620990 12859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:37.579120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32875 (local address 127.0.98.1:50927)
0927 12:48:37.586658 (+ 7538us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:37.586749 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0927 12:48:37.588386 (+ 1637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:48:37.594925 (+ 6539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:48:37.594938 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:48:37.595367 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:48:37.596165 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:37.596193 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:37.597553 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:37.597557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:37.599038 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:37.599051 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:37.602956 (+ 3905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:37.602961 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:37.603230 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:37.609905 (+ 6675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:48:37.609935 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:48:37.613730 (+ 3795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:48:37.616384 (+ 2654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:48:37.616393 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:48:37.616416 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:48:37.616875 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:48:37.617262 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:48:37.617265 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:48:37.617268 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:48:37.617491 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:48:37.618079 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:48:37.618087 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:48:37.618494 (+ 407us) client_negotiation.cc:769] Sending connection context
0927 12:48:37.619422 (+ 928us) client_negotiation.cc:240] Negotiation successful
0927 12:48:37.619828 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7244,"thread_start_us":129,"threads_started":1}
I20260927 12:48:37.622562 12856 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32875
I20260927 12:48:37.622855 12856 heartbeater.cc:464] Registering TS with master...
I20260927 12:48:37.623499 12856 heartbeater.cc:511] Master 127.0.98.62:32875 requested a full tablet report, sending...
I20260927 12:48:37.625393 12641 ts_manager.cc:194] Registered new tserver with Master: c3de391db8bc406383d179ec7ab6e5f5 (127.0.98.1:32911)
I20260927 12:48:37.627043 12641 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50927
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 12:48:37.646126 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:55541/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32875
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 12:48:37 dist-test-slave-lkn4 krb5kdc[12166](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790513317, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 12:48:37.815961 12862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:48:37.816334 12862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:48:37.816411 12862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:48:37.822718 12862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:48:37.822806 12862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:48:37.822886 12862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:48:37.831147 12862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:55541/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12862
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:48:37.832624 12862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:48:37.833992 12862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:48:37.847178 12869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:48:37.847559 12871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:48:37.848084 12868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:48:37.855680 12862 server_base.cc:1034] running on GCE node
I20260927 12:48:37.856384 12862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:48:37.857218 12862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:48:37.872750 12862 hybrid_clock.cc:648] HybridClock initialized: now 1790513317868812 us; error 3933 us; skew 500 ppm
Sep 27 12:48:37 dist-test-slave-lkn4 krb5kdc[12166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513317, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:48:37.877009 12862 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 12:48:37.878269 12862 webserver.cc:460] Webserver started at http://127.0.98.2:36893/ using document root <none> and password file <none>
I20260927 12:48:37.879005 12862 fs_manager.cc:359] Metadata directory not provided
I20260927 12:48:37.879092 12862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:48:37.879340 12862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 12:48:37 dist-test-slave-lkn4 krb5kdc[12166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513317, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 12:48:37.896034 12862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-1/data/instance:
uuid: "c6095c7c11104951be65e0b4c1c6c2f3"
format_stamp: "Formatted at 2026-09-27 12:48:37 on dist-test-slave-lkn4"
server_key: "4e0096fe9c4a8772a65de723715b38c6"
server_key_iv: "38ed9139b781c532f12c21267a8bc0c0"
server_key_version: "kuduclusterkey@0"
I20260927 12:48:37.896778 12862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c6095c7c11104951be65e0b4c1c6c2f3"
format_stamp: "Formatted at 2026-09-27 12:48:37 on dist-test-slave-lkn4"
server_key: "4e0096fe9c4a8772a65de723715b38c6"
server_key_iv: "38ed9139b781c532f12c21267a8bc0c0"
server_key_version: "kuduclusterkey@0"
I20260927 12:48:37.904649 12862 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20260927 12:48:37.941004 12878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:48:37.943436 12862 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.002s sys 0.000s
I20260927 12:48:37.943770 12862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-1/wal
uuid: "c6095c7c11104951be65e0b4c1c6c2f3"
format_stamp: "Formatted at 2026-09-27 12:48:37 on dist-test-slave-lkn4"
server_key: "4e0096fe9c4a8772a65de723715b38c6"
server_key_iv: "38ed9139b781c532f12c21267a8bc0c0"
server_key_version: "kuduclusterkey@0"
I20260927 12:48:37.944077 12862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:48:37.993184 12862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:48:37.996325 12862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:48:37.996506 12862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:48:37.997116 12862 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:48:37.998000 12862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:48:37.998071 12862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:48:37.998133 12862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:48:37.998171 12862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:48:38.008167 12862 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46653
I20260927 12:48:38.008183 12991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46653 every 8 connection(s)
I20260927 12:48:38.009159 12862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:48:38.018875 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 12862
I20260927 12:48:38.019002 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:48:38.019337 392 external_mini_cluster.cc:1428] Setting key 642abcd4b660ad588c77cd095b7112ec
Sep 27 12:48:38 dist-test-slave-lkn4 krb5kdc[12166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513317, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:48:38.039129 12858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:38.014478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42965 (local address 127.0.98.62:32875)
0927 12:48:38.014652 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 12:48:38.014657 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:48:38.021577 (+ 6920us) server_negotiation.cc:407] Connection header received
0927 12:48:38.022974 (+ 1397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:48:38.022978 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:48:38.023031 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:48:38.023081 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:48:38.024935 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:38.025835 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:38.027597 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:38.027963 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:38.028111 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:38.033798 (+ 5687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:48:38.033803 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:48:38.033806 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:48:38.033848 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:48:38.036201 (+ 2353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:48:38.037060 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:48:38.037064 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:48:38.037066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:48:38.037164 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:48:38.037614 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:48:38.037617 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:48:38.037619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:48:38.037865 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:48:38.037950 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 12:48:38.038754 (+ 804us) server_negotiation.cc:299] Negotiation successful
0927 12:48:38.038910 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260927 12:48:38.041174 12994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:38.011645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32875 (local address 127.0.98.2:42965)
0927 12:48:38.021371 (+ 9726us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:38.021464 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0927 12:48:38.022706 (+ 1242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:48:38.023537 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:48:38.023549 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:48:38.023939 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:48:38.024749 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:38.024772 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:38.026003 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:38.026006 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:38.027455 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:38.027466 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:38.028412 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:38.028415 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:38.028623 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:38.031724 (+ 3101us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:48:38.031749 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:48:38.033542 (+ 1793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:48:38.036487 (+ 2945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:48:38.036495 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:48:38.036516 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:48:38.036925 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:48:38.037275 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:48:38.037278 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:48:38.037280 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:48:38.037499 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:48:38.038021 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:48:38.038029 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:48:38.038450 (+ 421us) client_negotiation.cc:769] Sending connection context
0927 12:48:38.039402 (+ 952us) client_negotiation.cc:240] Negotiation successful
0927 12:48:38.039802 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9401,"thread_start_us":150,"threads_started":1}
I20260927 12:48:38.043108 12992 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32875
I20260927 12:48:38.043408 12992 heartbeater.cc:464] Registering TS with master...
I20260927 12:48:38.044147 12992 heartbeater.cc:511] Master 127.0.98.62:32875 requested a full tablet report, sending...
I20260927 12:48:38.045392 12641 ts_manager.cc:194] Registered new tserver with Master: c6095c7c11104951be65e0b4c1c6c2f3 (127.0.98.2:46653)
I20260927 12:48:38.046198 12641 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42965
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 12:48:38.077361 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:55541/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32875
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 12:48:38 dist-test-slave-lkn4 krb5kdc[12166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513317, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260927 12:48:38.229434 12997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:48:38.229836 12997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:48:38.229911 12997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:48:38.235895 12997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:48:38.235980 12997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:48:38.236061 12997 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:48:38.243981 12997 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:55541/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12997
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:48:38.245487 12997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:48:38.246918 12997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:48:38.259606 13005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:48:38.259603 13002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:48:38.260042 13003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:48:38.260360 12997 server_base.cc:1034] running on GCE node
I20260927 12:48:38.260850 12997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:48:38.261667 12997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:48:38.262940 12997 hybrid_clock.cc:648] HybridClock initialized: now 1790513318262896 us; error 60 us; skew 500 ppm
Sep 27 12:48:38 dist-test-slave-lkn4 krb5kdc[12166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513318, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:48:38.266583 12997 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 12:48:38.267766 12997 webserver.cc:460] Webserver started at http://127.0.98.3:37443/ using document root <none> and password file <none>
I20260927 12:48:38.268527 12997 fs_manager.cc:359] Metadata directory not provided
I20260927 12:48:38.268611 12997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:48:38.268848 12997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 12:48:38 dist-test-slave-lkn4 krb5kdc[12166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513318, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 12:48:38.291451 12997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-2/data/instance:
uuid: "7d7f6633fc3645af8996eca5883e205e"
format_stamp: "Formatted at 2026-09-27 12:48:38 on dist-test-slave-lkn4"
server_key: "77a2cdf6dae5af1fda907661e5521b08"
server_key_iv: "f19e06c72e17169cbbd8f9da10de020f"
server_key_version: "kuduclusterkey@0"
I20260927 12:48:38.292268 12997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7d7f6633fc3645af8996eca5883e205e"
format_stamp: "Formatted at 2026-09-27 12:48:38 on dist-test-slave-lkn4"
server_key: "77a2cdf6dae5af1fda907661e5521b08"
server_key_iv: "f19e06c72e17169cbbd8f9da10de020f"
server_key_version: "kuduclusterkey@0"
I20260927 12:48:38.298266 12997 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260927 12:48:38.316947 13012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:48:38.318828 12997 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 12:48:38.318969 12997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-2/wal
uuid: "7d7f6633fc3645af8996eca5883e205e"
format_stamp: "Formatted at 2026-09-27 12:48:38 on dist-test-slave-lkn4"
server_key: "77a2cdf6dae5af1fda907661e5521b08"
server_key_iv: "f19e06c72e17169cbbd8f9da10de020f"
server_key_version: "kuduclusterkey@0"
I20260927 12:48:38.319099 12997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:48:38.345994 12997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:48:38.349936 12997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:48:38.350225 12997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:48:38.351141 12997 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:48:38.352265 12997 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:48:38.352384 12997 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:48:38.352487 12997 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:48:38.352545 12997 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:48:38.363575 12997 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43535
I20260927 12:48:38.363618 13125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43535 every 8 connection(s)
I20260927 12:48:38.365000 12997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:48:38.369560 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 12997
I20260927 12:48:38.369693 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:48:38.369983 392 external_mini_cluster.cc:1428] Setting key 5d88e7dcf0cf8535f0ba5c4bcf783122
Sep 27 12:48:38 dist-test-slave-lkn4 krb5kdc[12166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513318, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:48:38.384584 12858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:38.369159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57265 (local address 127.0.98.62:32875)
0927 12:48:38.369344 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 12:48:38.369349 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:48:38.371031 (+ 1682us) server_negotiation.cc:407] Connection header received
0927 12:48:38.372277 (+ 1246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:48:38.372281 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:48:38.372335 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:48:38.372404 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:48:38.374133 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:38.374986 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:38.376687 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:38.377055 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:38.377173 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:38.380266 (+ 3093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:48:38.380270 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:48:38.380272 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:48:38.380305 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:48:38.381980 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:48:38.382648 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:48:38.382652 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:48:38.382654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:48:38.382756 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:48:38.383157 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:48:38.383160 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:48:38.383162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:48:38.383385 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:48:38.383462 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 12:48:38.384283 (+ 821us) server_negotiation.cc:299] Negotiation successful
0927 12:48:38.384419 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 12:48:38.386350 13128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:38.368012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32875 (local address 127.0.98.3:57265)
0927 12:48:38.370849 (+ 2837us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:38.370927 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0927 12:48:38.372028 (+ 1101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:48:38.372867 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:48:38.372878 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:48:38.373238 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:48:38.373932 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:38.373951 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:38.375130 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:38.375134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:38.376539 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:38.376551 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:38.377221 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:38.377225 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:38.377432 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:38.378328 (+ 896us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:48:38.378351 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:48:38.380058 (+ 1707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:48:38.382152 (+ 2094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:48:38.382158 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:48:38.382178 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:48:38.382532 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:48:38.382853 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:48:38.382856 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:48:38.382858 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:48:38.383051 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:48:38.383527 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:48:38.383533 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:48:38.383985 (+ 452us) client_negotiation.cc:769] Sending connection context
0927 12:48:38.384818 (+ 833us) client_negotiation.cc:240] Negotiation successful
0927 12:48:38.385193 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2338,"thread_start_us":153,"threads_started":1}
I20260927 12:48:38.387656 13126 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32875
I20260927 12:48:38.387952 13126 heartbeater.cc:464] Registering TS with master...
I20260927 12:48:38.388561 13126 heartbeater.cc:511] Master 127.0.98.62:32875 requested a full tablet report, sending...
I20260927 12:48:38.389786 12641 ts_manager.cc:194] Registered new tserver with Master: 7d7f6633fc3645af8996eca5883e205e (127.0.98.3:43535)
I20260927 12:48:38.390615 12641 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57265
I20260927 12:48:38.396106 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 12:48:38.396953 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 12723
I20260927 12:48:38.403170 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 12862
I20260927 12:48:38.409781 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 12997
I20260927 12:48:38.417991 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 12610
I20260927 12:48:38.425565 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32875
--webserver_interface=127.0.98.62
--webserver_port=36461
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32875
--ranger_config_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:55541/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:55541/kms with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:48:38.536214 13129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:48:38.536497 13129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:48:38.536542 13129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:48:38.540643 13129 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:48:38.540715 13129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:48:38.540733 13129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:48:38.540748 13129 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:48:38.540763 13129 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:48:38.546372 13129 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:55541/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32875
--ranger_config_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=36461
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:48:38.547436 13129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:48:38.548461 13129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:48:38.557214 13137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:48:38.557179 13134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:48:38.557192 13129 server_base.cc:1034] running on GCE node
W20260927 12:48:38.557163 13135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:48:38.558012 13129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:48:38.559171 13129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:48:38.560340 13129 hybrid_clock.cc:648] HybridClock initialized: now 1790513318560315 us; error 31 us; skew 500 ppm
Sep 27 12:48:38 dist-test-slave-lkn4 krb5kdc[12166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513318, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:48:38.563148 13129 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:48:38.564109 13129 webserver.cc:460] Webserver started at http://127.0.98.62:36461/ using document root <none> and password file <none>
I20260927 12:48:38.564637 13129 fs_manager.cc:359] Metadata directory not provided
I20260927 12:48:38.564710 13129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 12:48:38 dist-test-slave-lkn4 krb5kdc[12166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513318, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 12:48:38.632575 13129 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 12:48:38.635056 13145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:48:38.636493 13129 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 12:48:38.636611 13129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/master-0/wal
uuid: "125e81691bcc4d5f860760aa8a45362c"
format_stamp: "Formatted at 2026-09-27 12:48:36 on dist-test-slave-lkn4"
server_key: "8d0b6aa75498b459f26bc5431b85351b"
server_key_iv: "54a2ce4beaa4c7e257f0cade0ee912cb"
server_key_version: "kuduclusterkey@0"
I20260927 12:48:38.637204 13129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:48:38.651683 13129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:48:38.654901 13129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:48:38.655077 13129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:48:38.662300 13129 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32875
I20260927 12:48:38.662345 13197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32875 every 8 connection(s)
I20260927 12:48:38.663420 13129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:48:38.667770 13198 sys_catalog.cc:263] Verifying existing consensus state
I20260927 12:48:38.669430 13198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 125e81691bcc4d5f860760aa8a45362c: Bootstrap starting.
I20260927 12:48:38.673712 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 13129
I20260927 12:48:38.674139 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:55541/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:32911
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:32875
--webserver_port=44437
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:55541/kms with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 12:48:38.686275 13198 log.cc:824] T 00000000000000000000000000000000 P 125e81691bcc4d5f860760aa8a45362c: Log is configured to *not* fsync() on all Append() calls
I20260927 12:48:38.694963 13198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 125e81691bcc4d5f860760aa8a45362c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 12:48:38.695364 13198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 125e81691bcc4d5f860760aa8a45362c: Bootstrap complete.
I20260927 12:48:38.699452 13198 raft_consensus.cc:348] T 00000000000000000000000000000000 P 125e81691bcc4d5f860760aa8a45362c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "125e81691bcc4d5f860760aa8a45362c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32875 } }
I20260927 12:48:38.699849 13198 raft_consensus.cc:729] T 00000000000000000000000000000000 P 125e81691bcc4d5f860760aa8a45362c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 125e81691bcc4d5f860760aa8a45362c, State: Initialized, Role: FOLLOWER
I20260927 12:48:38.700327 13198 consensus_queue.cc:260] T 00000000000000000000000000000000 P 125e81691bcc4d5f860760aa8a45362c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "125e81691bcc4d5f860760aa8a45362c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32875 } }
I20260927 12:48:38.700446 13198 raft_consensus.cc:388] T 00000000000000000000000000000000 P 125e81691bcc4d5f860760aa8a45362c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:48:38.700511 13198 raft_consensus.cc:482] T 00000000000000000000000000000000 P 125e81691bcc4d5f860760aa8a45362c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:48:38.700603 13198 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 125e81691bcc4d5f860760aa8a45362c [term 1 FOLLOWER]: Advancing to term 2
I20260927 12:48:38.702535 13198 raft_consensus.cc:504] T 00000000000000000000000000000000 P 125e81691bcc4d5f860760aa8a45362c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "125e81691bcc4d5f860760aa8a45362c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32875 } }
I20260927 12:48:38.702890 13198 leader_election.cc:302] T 00000000000000000000000000000000 P 125e81691bcc4d5f860760aa8a45362c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 125e81691bcc4d5f860760aa8a45362c; no voters:
I20260927 12:48:38.703182 13198 leader_election.cc:288] T 00000000000000000000000000000000 P 125e81691bcc4d5f860760aa8a45362c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 12:48:38.703329 13202 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 125e81691bcc4d5f860760aa8a45362c [term 2 FOLLOWER]: Leader election won for term 2
I20260927 12:48:38.703603 13202 raft_consensus.cc:686] T 00000000000000000000000000000000 P 125e81691bcc4d5f860760aa8a45362c [term 2 LEADER]: Becoming Leader. State: Replica: 125e81691bcc4d5f860760aa8a45362c, State: Running, Role: LEADER
I20260927 12:48:38.703981 13202 consensus_queue.cc:237] T 00000000000000000000000000000000 P 125e81691bcc4d5f860760aa8a45362c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "125e81691bcc4d5f860760aa8a45362c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32875 } }
I20260927 12:48:38.704458 13198 sys_catalog.cc:564] T 00000000000000000000000000000000 P 125e81691bcc4d5f860760aa8a45362c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:48:38.705041 13202 sys_catalog.cc:455] T 00000000000000000000000000000000 P 125e81691bcc4d5f860760aa8a45362c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 125e81691bcc4d5f860760aa8a45362c. Latest consensus state: current_term: 2 leader_uuid: "125e81691bcc4d5f860760aa8a45362c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "125e81691bcc4d5f860760aa8a45362c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32875 } } }
I20260927 12:48:38.705150 13202 sys_catalog.cc:458] T 00000000000000000000000000000000 P 125e81691bcc4d5f860760aa8a45362c [sys.catalog]: This master's current role is: LEADER
I20260927 12:48:38.705834 13203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 125e81691bcc4d5f860760aa8a45362c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "125e81691bcc4d5f860760aa8a45362c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "125e81691bcc4d5f860760aa8a45362c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32875 } } }
I20260927 12:48:38.705945 13203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 125e81691bcc4d5f860760aa8a45362c [sys.catalog]: This master's current role is: LEADER
I20260927 12:48:38.706513 13206 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:48:38.708343 13198 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 12:48:38.709586 13206 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:48:38.712813 13206 catalog_manager.cc:1259] Loaded cluster ID: 21710f5edef2472cb74025cc1b64cd7c
I20260927 12:48:38.712941 13206 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:48:38.716593 13206 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:48:38.717571 13206 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 125e81691bcc4d5f860760aa8a45362c: Loaded TSK: 0
I20260927 12:48:38.718969 13206 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 12:48:38.797441 13200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:48:38.797701 13200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:48:38.797760 13200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:48:38.802349 13200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:48:38.802412 13200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:48:38.802489 13200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:48:38.808292 13200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:55541/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:32911
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=44437
--tserver_master_addrs=127.0.98.62:32875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:48:38.809449 13200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:48:38.810563 13200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:48:38.822738 13238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:48:38.822739 13236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:48:38.823197 13235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:48:38.823778 13200 server_base.cc:1034] running on GCE node
I20260927 12:48:38.824211 13200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:48:38.824913 13200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:48:38.826088 13200 hybrid_clock.cc:648] HybridClock initialized: now 1790513318826054 us; error 43 us; skew 500 ppm
Sep 27 12:48:38 dist-test-slave-lkn4 krb5kdc[12166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513318, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:48:38.828912 13200 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 12:48:38.829941 13200 webserver.cc:460] Webserver started at http://127.0.98.1:44437/ using document root <none> and password file <none>
I20260927 12:48:38.830549 13200 fs_manager.cc:359] Metadata directory not provided
I20260927 12:48:38.830631 13200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 12:48:38 dist-test-slave-lkn4 krb5kdc[12166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513318, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 12:48:38.855805 13200 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.003s sys 0.000s
I20260927 12:48:38.858642 13246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:48:38.859952 13200 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:48:38.860069 13200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-0/wal
uuid: "c3de391db8bc406383d179ec7ab6e5f5"
format_stamp: "Formatted at 2026-09-27 12:48:37 on dist-test-slave-lkn4"
server_key: "845d5ab1f76e9a47dd25b60fe5fe78cb"
server_key_iv: "e372c974abac3612311e0226b8e4faad"
server_key_version: "kuduclusterkey@0"
I20260927 12:48:38.860621 13200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:48:38.873152 13200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:48:38.876384 13200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:48:38.876586 13200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:48:38.877315 13200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:48:38.878383 13200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:48:38.878443 13200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:48:38.878500 13200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:48:38.878523 13200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:48:38.889269 13200 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:32911
I20260927 12:48:38.889284 13359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:32911 every 8 connection(s)
I20260927 12:48:38.891100 13200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:48:38.893323 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 13200
I20260927 12:48:38.893791 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:55541/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:46653
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:32875
--webserver_port=36893
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:55541/kms with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 12:48:38 dist-test-slave-lkn4 krb5kdc[12166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513318, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:48:38.916095 13364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:38.894865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38175 (local address 127.0.98.62:32875)
0927 12:48:38.898359 (+ 3494us) server_negotiation.cc:206] Beginning negotiation
0927 12:48:38.898382 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 12:48:38.898609 (+ 227us) server_negotiation.cc:407] Connection header received
0927 12:48:38.900188 (+ 1579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:48:38.900206 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:48:38.900618 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:48:38.900955 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:48:38.902858 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:38.904343 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:38.906079 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:38.906814 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:38.906969 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:38.910169 (+ 3200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:48:38.910176 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:48:38.910195 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:48:38.910228 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:48:38.912195 (+ 1967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:48:38.912848 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:48:38.912854 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:48:38.912860 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:48:38.913003 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:48:38.913375 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:48:38.913378 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:48:38.913380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:48:38.913860 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:48:38.913964 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 12:48:38.914735 (+ 771us) server_negotiation.cc:299] Negotiation successful
0927 12:48:38.915067 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3299,"thread_start_us":115,"threads_started":1}
I20260927 12:48:38.918241 13362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:38.893583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32875 (local address 127.0.98.1:38175)
0927 12:48:38.898403 (+ 4820us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:38.898468 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0927 12:48:38.899661 (+ 1193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:48:38.901576 (+ 1915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:48:38.901586 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:48:38.901944 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:48:38.902652 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:38.902673 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:38.904515 (+ 1842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:38.904519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:38.905909 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:38.905920 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:38.907247 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:38.907251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:38.907456 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:38.908086 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:48:38.908108 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:48:38.909879 (+ 1771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:48:38.912330 (+ 2451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:48:38.912337 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:48:38.912353 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:48:38.912705 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:48:38.913111 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:48:38.913114 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:48:38.913116 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:48:38.913278 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:48:38.914070 (+ 792us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:48:38.914077 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:48:38.914391 (+ 314us) client_negotiation.cc:769] Sending connection context
0927 12:48:38.916875 (+ 2484us) client_negotiation.cc:240] Negotiation successful
0927 12:48:38.917226 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4576,"thread_start_us":134,"threads_started":1}
I20260927 12:48:38.921607 13360 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32875
I20260927 12:48:38.921890 13360 heartbeater.cc:464] Registering TS with master...
I20260927 12:48:38.922441 13360 heartbeater.cc:511] Master 127.0.98.62:32875 requested a full tablet report, sending...
W20260927 12:48:38.924734 13360 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:32875 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 12:48:38.924868 13360 heartbeater.cc:464] Registering TS with master...
I20260927 12:48:38.925043 13360 heartbeater.cc:511] Master 127.0.98.62:32875 requested a full tablet report, sending...
I20260927 12:48:38.925969 13360 heartbeater.cc:464] Registering TS with master...
I20260927 12:48:38.926195 13360 heartbeater.cc:511] Master 127.0.98.62:32875 requested a full tablet report, sending...
W20260927 12:48:38.926997 13360 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 12:48:39.046451 13363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:48:39.046728 13363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:48:39.046773 13363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:48:39.051498 13363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:48:39.051592 13363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:48:39.051685 13363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:48:39.059893 13363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:55541/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:46653
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36893
--tserver_master_addrs=127.0.98.62:32875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:48:39.061424 13363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:48:39.062826 13363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:48:39.074288 13372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:48:39.074288 13369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:48:39.074667 13370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:48:39.074681 13363 server_base.cc:1034] running on GCE node
I20260927 12:48:39.075161 13363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:48:39.075943 13363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:48:39.077113 13363 hybrid_clock.cc:648] HybridClock initialized: now 1790513319077073 us; error 60 us; skew 500 ppm
Sep 27 12:48:39 dist-test-slave-lkn4 krb5kdc[12166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513319, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:48:39.080082 13363 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 12:48:39.081055 13363 webserver.cc:460] Webserver started at http://127.0.98.2:36893/ using document root <none> and password file <none>
I20260927 12:48:39.081619 13363 fs_manager.cc:359] Metadata directory not provided
I20260927 12:48:39.081697 13363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 12:48:39 dist-test-slave-lkn4 krb5kdc[12166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513319, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 12:48:39.101397 13363 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 12:48:39.104713 13380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:48:39.106190 13363 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:48:39.106379 13363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-1/wal
uuid: "c6095c7c11104951be65e0b4c1c6c2f3"
format_stamp: "Formatted at 2026-09-27 12:48:37 on dist-test-slave-lkn4"
server_key: "4e0096fe9c4a8772a65de723715b38c6"
server_key_iv: "38ed9139b781c532f12c21267a8bc0c0"
server_key_version: "kuduclusterkey@0"
I20260927 12:48:39.107137 13363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:48:39.117846 13363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:48:39.120797 13363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:48:39.120950 13363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:48:39.121502 13363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:48:39.122362 13363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:48:39.122400 13363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:48:39.122439 13363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:48:39.122453 13363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:48:39.134841 13363 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46653
I20260927 12:48:39.136281 13363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:48:39.137089 13493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46653 every 8 connection(s)
I20260927 12:48:39.145210 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 13363
I20260927 12:48:39.145695 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:55541/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:43535
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:32875
--webserver_port=37443
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:55541/kms with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 12:48:39 dist-test-slave-lkn4 krb5kdc[12166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513319, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:48:39.152320 13364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:39.138436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52961 (local address 127.0.98.62:32875)
0927 12:48:39.138569 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 12:48:39.138574 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:48:39.139557 (+ 983us) server_negotiation.cc:407] Connection header received
0927 12:48:39.140682 (+ 1125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:48:39.140685 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:48:39.140752 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:48:39.140831 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:48:39.142012 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:39.142663 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:39.144175 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:39.144544 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:39.144622 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:39.147937 (+ 3315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:48:39.147941 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:48:39.147944 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:48:39.147978 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:48:39.149706 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:48:39.150388 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:48:39.150392 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:48:39.150394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:48:39.150494 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:48:39.150914 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:48:39.150918 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:48:39.150920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:48:39.151169 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:48:39.151273 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 12:48:39.151959 (+ 686us) server_negotiation.cc:299] Negotiation successful
0927 12:48:39.152122 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 12:48:39.153388 13496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:39.138936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32875 (local address 127.0.98.2:52961)
0927 12:48:39.139398 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:39.139451 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 12:48:39.140432 (+ 981us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:48:39.141012 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:48:39.141020 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:48:39.141342 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:48:39.141873 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:39.141888 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:39.142786 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:39.142789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:39.144029 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:39.144040 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:39.144883 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:39.144888 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:39.145118 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:39.145818 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:48:39.145845 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:48:39.147683 (+ 1838us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:48:39.149858 (+ 2175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:48:39.149865 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:48:39.149884 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:48:39.150280 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:48:39.150600 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:48:39.150603 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:48:39.150605 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:48:39.150805 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:48:39.151281 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:48:39.151288 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:48:39.151656 (+ 368us) client_negotiation.cc:769] Sending connection context
0927 12:48:39.151899 (+ 243us) client_negotiation.cc:240] Negotiation successful
0927 12:48:39.152263 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":113,"threads_started":1}
I20260927 12:48:39.154788 13494 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32875
I20260927 12:48:39.155061 13494 heartbeater.cc:464] Registering TS with master...
I20260927 12:48:39.155656 13494 heartbeater.cc:511] Master 127.0.98.62:32875 requested a full tablet report, sending...
W20260927 12:48:39.157490 13494 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:32875 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 12:48:39.157605 13494 heartbeater.cc:464] Registering TS with master...
I20260927 12:48:39.157786 13494 heartbeater.cc:511] Master 127.0.98.62:32875 requested a full tablet report, sending...
I20260927 12:48:39.158952 13494 heartbeater.cc:464] Registering TS with master...
I20260927 12:48:39.159129 13494 heartbeater.cc:511] Master 127.0.98.62:32875 requested a full tablet report, sending...
W20260927 12:48:39.159826 13494 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 12:48:39.297722 13497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:48:39.298029 13497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:48:39.298110 13497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:48:39.303943 13497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:48:39.304035 13497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:48:39.304123 13497 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:48:39.310278 13497 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:55541/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:43535
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=37443
--tserver_master_addrs=127.0.98.62:32875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13497
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:48:39.311439 13497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:48:39.312561 13497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:48:39.322393 13506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:48:39.322882 13503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:48:39.323145 13504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:48:39.327168 13497 server_base.cc:1034] running on GCE node
I20260927 12:48:39.327843 13497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:48:39.328650 13497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:48:39.329826 13497 hybrid_clock.cc:648] HybridClock initialized: now 1790513319329796 us; error 43 us; skew 500 ppm
Sep 27 12:48:39 dist-test-slave-lkn4 krb5kdc[12166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513319, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:48:39.332567 13497 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 12:48:39.333482 13497 webserver.cc:460] Webserver started at http://127.0.98.3:37443/ using document root <none> and password file <none>
I20260927 12:48:39.334112 13497 fs_manager.cc:359] Metadata directory not provided
I20260927 12:48:39.334200 13497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 12:48:39 dist-test-slave-lkn4 krb5kdc[12166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513319, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 12:48:39.356004 13497 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.000s sys 0.003s
I20260927 12:48:39.363847 13514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:48:39.365871 13497 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260927 12:48:39.366034 13497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-2/wal
uuid: "7d7f6633fc3645af8996eca5883e205e"
format_stamp: "Formatted at 2026-09-27 12:48:38 on dist-test-slave-lkn4"
server_key: "77a2cdf6dae5af1fda907661e5521b08"
server_key_iv: "f19e06c72e17169cbbd8f9da10de020f"
server_key_version: "kuduclusterkey@0"
I20260927 12:48:39.366885 13497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:48:39.385612 13497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:48:39.389546 13497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:48:39.389755 13497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:48:39.390569 13497 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:48:39.391687 13497 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:48:39.391749 13497 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:48:39.391800 13497 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:48:39.391830 13497 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:48:39.403019 13497 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43535
I20260927 12:48:39.403079 13627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43535 every 8 connection(s)
I20260927 12:48:39.404376 13497 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:48:39.408890 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 13497
Sep 27 12:48:39 dist-test-slave-lkn4 krb5kdc[12166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513319, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:48:39.421556 13631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:39.409511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59482 (local address 127.0.98.62:32875)
0927 12:48:39.409956 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0927 12:48:39.409960 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:48:39.412206 (+ 2246us) server_negotiation.cc:407] Connection header received
0927 12:48:39.412241 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:48:39.412244 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:48:39.412301 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:48:39.412390 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:48:39.414132 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:39.414939 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:39.416218 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:39.416578 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:39.416696 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:39.418368 (+ 1672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:48:39.418371 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:48:39.418373 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:48:39.418418 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:48:39.419925 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:48:39.420316 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:48:39.420319 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:48:39.420321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:48:39.420396 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:48:39.420630 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:48:39.420633 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:48:39.420635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:48:39.420823 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:48:39.421010 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0927 12:48:39.421208 (+ 198us) server_negotiation.cc:299] Negotiation successful
0927 12:48:39.421365 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":86,"threads_started":1}
I20260927 12:48:39.425221 13364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:39.406941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33321 (local address 127.0.98.62:32875)
0927 12:48:39.407097 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 12:48:39.407102 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:48:39.408248 (+ 1146us) server_negotiation.cc:407] Connection header received
0927 12:48:39.410075 (+ 1827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:48:39.410079 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:48:39.410129 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:48:39.410201 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:48:39.412221 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:39.412941 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:39.417882 (+ 4941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:39.418235 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:39.418346 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:39.421200 (+ 2854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:48:39.421205 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:48:39.421207 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:48:39.421233 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:48:39.423061 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:48:39.423673 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:48:39.423675 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:48:39.423677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:48:39.423754 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:48:39.424141 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:48:39.424144 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:48:39.424146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:48:39.424327 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:48:39.424403 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 12:48:39.424942 (+ 539us) server_negotiation.cc:299] Negotiation successful
0927 12:48:39.425059 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57,"spinlock_wait_cycles":27008}
I20260927 12:48:39.426206 13630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:39.407490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32875 (local address 127.0.98.3:33321)
0927 12:48:39.408075 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:39.408134 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 12:48:39.409844 (+ 1710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:48:39.410915 (+ 1071us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:48:39.410923 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:48:39.411559 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:48:39.412073 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:39.412089 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:39.413065 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:39.413069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:39.417741 (+ 4672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:39.417755 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:39.418545 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:39.418549 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:39.418771 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:39.419371 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:48:39.419394 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:48:39.421001 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:48:39.423183 (+ 2182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:48:39.423189 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:48:39.423209 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:48:39.423565 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:48:39.423848 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:48:39.423851 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:48:39.423854 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:48:39.424031 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:48:39.424431 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:48:39.424438 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:48:39.424714 (+ 276us) client_negotiation.cc:769] Sending connection context
0927 12:48:39.424892 (+ 178us) client_negotiation.cc:240] Negotiation successful
0927 12:48:39.425218 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":144,"threads_started":1}
I20260927 12:48:39.427254 13628 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32875
I20260927 12:48:39.427495 13628 heartbeater.cc:464] Registering TS with master...
I20260927 12:48:39.427984 13628 heartbeater.cc:511] Master 127.0.98.62:32875 requested a full tablet report, sending...
W20260927 12:48:39.429420 13628 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:32875 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 12:48:39.429530 13628 heartbeater.cc:464] Registering TS with master...
I20260927 12:48:39.429687 13628 heartbeater.cc:511] Master 127.0.98.62:32875 requested a full tablet report, sending...
I20260927 12:48:39.430475 13628 heartbeater.cc:464] Registering TS with master...
I20260927 12:48:39.430634 13628 heartbeater.cc:511] Master 127.0.98.62:32875 requested a full tablet report, sending...
W20260927 12:48:39.431360 13628 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 12:48:39 dist-test-slave-lkn4 krb5kdc[12166](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790513319, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:48:39.927238 13360 heartbeater.cc:464] Registering TS with master...
I20260927 12:48:39.927538 13360 heartbeater.cc:511] Master 127.0.98.62:32875 requested a full tablet report, sending...
I20260927 12:48:39.928689 13162 ts_manager.cc:194] Registered new tserver with Master: c3de391db8bc406383d179ec7ab6e5f5 (127.0.98.1:32911)
I20260927 12:48:39.930284 13162 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38175
I20260927 12:48:40.160041 13494 heartbeater.cc:464] Registering TS with master...
I20260927 12:48:40.160342 13494 heartbeater.cc:511] Master 127.0.98.62:32875 requested a full tablet report, sending...
I20260927 12:48:40.161131 13162 ts_manager.cc:194] Registered new tserver with Master: c6095c7c11104951be65e0b4c1c6c2f3 (127.0.98.2:46653)
I20260927 12:48:40.161942 13162 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52961
Sep 27 12:48:40 dist-test-slave-lkn4 krb5kdc[12166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513319, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 12:48:40.431604 13628 heartbeater.cc:464] Registering TS with master...
I20260927 12:48:40.431929 13628 heartbeater.cc:511] Master 127.0.98.62:32875 requested a full tablet report, sending...
I20260927 12:48:40.432785 13162 ts_manager.cc:194] Registered new tserver with Master: 7d7f6633fc3645af8996eca5883e205e (127.0.98.3:43535)
I20260927 12:48:40.433601 13162 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33321
Sep 27 12:48:40 dist-test-slave-lkn4 krb5kdc[12166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513319, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 12:48:40.438512 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 12:48:40.458854 13651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:40.440037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59488 (local address 127.0.98.62:32875)
0927 12:48:40.440293 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0927 12:48:40.440298 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:48:40.440508 (+ 210us) server_negotiation.cc:407] Connection header received
0927 12:48:40.440545 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:48:40.440547 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:48:40.440590 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:48:40.440696 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:48:40.441402 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:40.442139 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:40.443302 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:40.443692 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:40.443806 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:40.445638 (+ 1832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:48:40.445642 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:48:40.445645 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:48:40.445678 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:48:40.456113 (+ 10435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:48:40.456902 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:48:40.456907 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:48:40.456909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:48:40.457024 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:48:40.457364 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:48:40.457368 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:48:40.457369 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:48:40.457625 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:48:40.457762 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 12:48:40.458523 (+ 761us) server_negotiation.cc:299] Negotiation successful
0927 12:48:40.458666 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":85,"threads_started":1}
I20260927 12:48:40.463620 13162 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59488:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:48:40.467144 13162 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:48:40.488723 13660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:40.481339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46653 (local address 127.0.0.1:52538)
0927 12:48:40.482883 (+ 1544us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:40.482896 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:48:40.482975 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:48:40.484096 (+ 1121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:48:40.484099 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:48:40.484117 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:48:40.484208 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:40.484215 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:40.485525 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:40.485528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:40.486790 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:40.486801 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:40.488110 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:40.488114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:40.488265 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:40.488314 (+ 49us) client_negotiation.cc:769] Sending connection context
0927 12:48:40.488418 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 12:48:40.488546 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1480,"mutex_wait_us":4,"thread_start_us":97,"threads_started":1}
I20260927 12:48:40.489152 13656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:40.479619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32911 (local address 127.0.0.1:58100)
0927 12:48:40.482169 (+ 2550us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:40.482232 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 12:48:40.482373 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:48:40.483769 (+ 1396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:48:40.483776 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:48:40.483799 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:48:40.483924 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:40.483930 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:40.485345 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:40.485348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:40.486856 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:40.486867 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:40.488113 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:40.488117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:40.488865 (+ 748us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:40.488888 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 12:48:40.488944 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 12:48:40.489008 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2333,"mutex_wait_us":680,"thread_start_us":123,"threads_started":1}
I20260927 12:48:40.489495 13657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:40.480486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58100 (local address 127.0.98.1:32911)
0927 12:48:40.483105 (+ 2619us) server_negotiation.cc:206] Beginning negotiation
0927 12:48:40.483112 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 12:48:40.483136 (+ 24us) server_negotiation.cc:407] Connection header received
0927 12:48:40.483224 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:48:40.483229 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:48:40.483445 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:48:40.483586 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:48:40.484047 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:40.485215 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:40.486993 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:40.487812 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:40.487912 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:40.488051 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 12:48:40.489258 (+ 1207us) server_negotiation.cc:299] Negotiation successful
0927 12:48:40.489360 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2512,"thread_start_us":136,"threads_started":1}
I20260927 12:48:40.490589 13659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:40.481737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52538 (local address 127.0.98.2:46653)
0927 12:48:40.483474 (+ 1737us) server_negotiation.cc:206] Beginning negotiation
0927 12:48:40.483481 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 12:48:40.483503 (+ 22us) server_negotiation.cc:407] Connection header received
0927 12:48:40.483578 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:48:40.483583 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:48:40.483793 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:48:40.483939 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:48:40.484354 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:40.485402 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:40.486918 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:40.487767 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:40.487875 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:40.488042 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 12:48:40.490347 (+ 2305us) server_negotiation.cc:299] Negotiation successful
0927 12:48:40.490452 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1635,"thread_start_us":86,"threads_started":1}
I20260927 12:48:40.493304 13276 tablet_service.cc:1467] Processing CreateTablet for tablet 932fc63ec8e947f78e3faa8726d7489d (DEFAULT_TABLE table=test-table [id=13e3becbd24f4154a3c8aaa970dbb26b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:48:40.493463 13411 tablet_service.cc:1467] Processing CreateTablet for tablet 932fc63ec8e947f78e3faa8726d7489d (DEFAULT_TABLE table=test-table [id=13e3becbd24f4154a3c8aaa970dbb26b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:48:40.494477 13276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 932fc63ec8e947f78e3faa8726d7489d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:48:40.494477 13411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 932fc63ec8e947f78e3faa8726d7489d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:48:40.509100 13658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:40.482528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43535 (local address 127.0.0.1:49632)
0927 12:48:40.483153 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:40.483165 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:48:40.483253 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:48:40.489698 (+ 6445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:48:40.489701 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:48:40.489730 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:48:40.489814 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:40.489820 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:40.503739 (+ 13919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:40.503743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:40.505178 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:40.505194 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:40.508636 (+ 3442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:40.508641 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:40.508761 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:40.508792 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 12:48:40.508844 (+ 52us) client_negotiation.cc:240] Negotiation successful
0927 12:48:40.508934 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":92,"threads_started":1}
I20260927 12:48:40.510280 13661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:40.482711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49632 (local address 127.0.98.3:43535)
0927 12:48:40.488382 (+ 5671us) server_negotiation.cc:206] Beginning negotiation
0927 12:48:40.488389 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 12:48:40.488413 (+ 24us) server_negotiation.cc:407] Connection header received
0927 12:48:40.488484 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:48:40.488490 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:48:40.488687 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:48:40.489971 (+ 1284us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:48:40.490166 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:40.503577 (+ 13411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:40.505374 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:40.506302 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:40.509633 (+ 3331us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:40.509813 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 12:48:40.510001 (+ 188us) server_negotiation.cc:299] Negotiation successful
0927 12:48:40.510130 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5557,"thread_start_us":73,"threads_started":1}
I20260927 12:48:40.511914 13663 tablet_bootstrap.cc:492] T 932fc63ec8e947f78e3faa8726d7489d P c3de391db8bc406383d179ec7ab6e5f5: Bootstrap starting.
I20260927 12:48:40.513686 13562 tablet_service.cc:1467] Processing CreateTablet for tablet 932fc63ec8e947f78e3faa8726d7489d (DEFAULT_TABLE table=test-table [id=13e3becbd24f4154a3c8aaa970dbb26b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:48:40.514798 13663 tablet_bootstrap.cc:654] T 932fc63ec8e947f78e3faa8726d7489d P c3de391db8bc406383d179ec7ab6e5f5: Neither blocks nor log segments found. Creating new log.
I20260927 12:48:40.514904 13562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 932fc63ec8e947f78e3faa8726d7489d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:48:40.515920 13663 log.cc:824] T 932fc63ec8e947f78e3faa8726d7489d P c3de391db8bc406383d179ec7ab6e5f5: Log is configured to *not* fsync() on all Append() calls
I20260927 12:48:40.518363 13663 tablet_bootstrap.cc:492] T 932fc63ec8e947f78e3faa8726d7489d P c3de391db8bc406383d179ec7ab6e5f5: No bootstrap required, opened a new log
I20260927 12:48:40.518586 13663 ts_tablet_manager.cc:1397] T 932fc63ec8e947f78e3faa8726d7489d P c3de391db8bc406383d179ec7ab6e5f5: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 12:48:40.523077 13663 raft_consensus.cc:348] T 932fc63ec8e947f78e3faa8726d7489d P c3de391db8bc406383d179ec7ab6e5f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3de391db8bc406383d179ec7ab6e5f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32911 } } peers { permanent_uuid: "c6095c7c11104951be65e0b4c1c6c2f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46653 } } peers { permanent_uuid: "7d7f6633fc3645af8996eca5883e205e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43535 } }
I20260927 12:48:40.523547 13663 raft_consensus.cc:374] T 932fc63ec8e947f78e3faa8726d7489d P c3de391db8bc406383d179ec7ab6e5f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:48:40.523664 13663 raft_consensus.cc:729] T 932fc63ec8e947f78e3faa8726d7489d P c3de391db8bc406383d179ec7ab6e5f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3de391db8bc406383d179ec7ab6e5f5, State: Initialized, Role: FOLLOWER
I20260927 12:48:40.524252 13663 consensus_queue.cc:260] T 932fc63ec8e947f78e3faa8726d7489d P c3de391db8bc406383d179ec7ab6e5f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3de391db8bc406383d179ec7ab6e5f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32911 } } peers { permanent_uuid: "c6095c7c11104951be65e0b4c1c6c2f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46653 } } peers { permanent_uuid: "7d7f6633fc3645af8996eca5883e205e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43535 } }
I20260927 12:48:40.525717 13665 tablet_bootstrap.cc:492] T 932fc63ec8e947f78e3faa8726d7489d P c6095c7c11104951be65e0b4c1c6c2f3: Bootstrap starting.
I20260927 12:48:40.525952 13663 ts_tablet_manager.cc:1428] T 932fc63ec8e947f78e3faa8726d7489d P c3de391db8bc406383d179ec7ab6e5f5: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260927 12:48:40.526392 13360 heartbeater.cc:503] Master 127.0.98.62:32875 was elected leader, sending a full tablet report...
I20260927 12:48:40.528240 13665 tablet_bootstrap.cc:654] T 932fc63ec8e947f78e3faa8726d7489d P c6095c7c11104951be65e0b4c1c6c2f3: Neither blocks nor log segments found. Creating new log.
I20260927 12:48:40.529971 13667 tablet_bootstrap.cc:492] T 932fc63ec8e947f78e3faa8726d7489d P 7d7f6633fc3645af8996eca5883e205e: Bootstrap starting.
I20260927 12:48:40.532306 13665 log.cc:824] T 932fc63ec8e947f78e3faa8726d7489d P c6095c7c11104951be65e0b4c1c6c2f3: Log is configured to *not* fsync() on all Append() calls
I20260927 12:48:40.532836 13667 tablet_bootstrap.cc:654] T 932fc63ec8e947f78e3faa8726d7489d P 7d7f6633fc3645af8996eca5883e205e: Neither blocks nor log segments found. Creating new log.
I20260927 12:48:40.533578 13667 log.cc:824] T 932fc63ec8e947f78e3faa8726d7489d P 7d7f6633fc3645af8996eca5883e205e: Log is configured to *not* fsync() on all Append() calls
I20260927 12:48:40.535943 13667 tablet_bootstrap.cc:492] T 932fc63ec8e947f78e3faa8726d7489d P 7d7f6633fc3645af8996eca5883e205e: No bootstrap required, opened a new log
I20260927 12:48:40.536263 13667 ts_tablet_manager.cc:1397] T 932fc63ec8e947f78e3faa8726d7489d P 7d7f6633fc3645af8996eca5883e205e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.002s
I20260927 12:48:40.540606 13667 raft_consensus.cc:348] T 932fc63ec8e947f78e3faa8726d7489d P 7d7f6633fc3645af8996eca5883e205e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3de391db8bc406383d179ec7ab6e5f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32911 } } peers { permanent_uuid: "c6095c7c11104951be65e0b4c1c6c2f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46653 } } peers { permanent_uuid: "7d7f6633fc3645af8996eca5883e205e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43535 } }
I20260927 12:48:40.541039 13667 raft_consensus.cc:374] T 932fc63ec8e947f78e3faa8726d7489d P 7d7f6633fc3645af8996eca5883e205e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:48:40.541157 13667 raft_consensus.cc:729] T 932fc63ec8e947f78e3faa8726d7489d P 7d7f6633fc3645af8996eca5883e205e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d7f6633fc3645af8996eca5883e205e, State: Initialized, Role: FOLLOWER
I20260927 12:48:40.541934 13667 consensus_queue.cc:260] T 932fc63ec8e947f78e3faa8726d7489d P 7d7f6633fc3645af8996eca5883e205e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3de391db8bc406383d179ec7ab6e5f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32911 } } peers { permanent_uuid: "c6095c7c11104951be65e0b4c1c6c2f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46653 } } peers { permanent_uuid: "7d7f6633fc3645af8996eca5883e205e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43535 } }
I20260927 12:48:40.543272 13667 ts_tablet_manager.cc:1428] T 932fc63ec8e947f78e3faa8726d7489d P 7d7f6633fc3645af8996eca5883e205e: Time spent starting tablet: real 0.007s user 0.004s sys 0.001s
I20260927 12:48:40.544385 13628 heartbeater.cc:503] Master 127.0.98.62:32875 was elected leader, sending a full tablet report...
I20260927 12:48:40.557253 13665 tablet_bootstrap.cc:492] T 932fc63ec8e947f78e3faa8726d7489d P c6095c7c11104951be65e0b4c1c6c2f3: No bootstrap required, opened a new log
I20260927 12:48:40.557507 13665 ts_tablet_manager.cc:1397] T 932fc63ec8e947f78e3faa8726d7489d P c6095c7c11104951be65e0b4c1c6c2f3: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.000s
I20260927 12:48:40.564054 13665 raft_consensus.cc:348] T 932fc63ec8e947f78e3faa8726d7489d P c6095c7c11104951be65e0b4c1c6c2f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3de391db8bc406383d179ec7ab6e5f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32911 } } peers { permanent_uuid: "c6095c7c11104951be65e0b4c1c6c2f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46653 } } peers { permanent_uuid: "7d7f6633fc3645af8996eca5883e205e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43535 } }
I20260927 12:48:40.564543 13665 raft_consensus.cc:374] T 932fc63ec8e947f78e3faa8726d7489d P c6095c7c11104951be65e0b4c1c6c2f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:48:40.564668 13665 raft_consensus.cc:729] T 932fc63ec8e947f78e3faa8726d7489d P c6095c7c11104951be65e0b4c1c6c2f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6095c7c11104951be65e0b4c1c6c2f3, State: Initialized, Role: FOLLOWER
I20260927 12:48:40.565385 13665 consensus_queue.cc:260] T 932fc63ec8e947f78e3faa8726d7489d P c6095c7c11104951be65e0b4c1c6c2f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3de391db8bc406383d179ec7ab6e5f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32911 } } peers { permanent_uuid: "c6095c7c11104951be65e0b4c1c6c2f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46653 } } peers { permanent_uuid: "7d7f6633fc3645af8996eca5883e205e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43535 } }
I20260927 12:48:40.566900 13665 ts_tablet_manager.cc:1428] T 932fc63ec8e947f78e3faa8726d7489d P c6095c7c11104951be65e0b4c1c6c2f3: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20260927 12:48:40.567390 13494 heartbeater.cc:503] Master 127.0.98.62:32875 was elected leader, sending a full tablet report...
W20260927 12:48:40.638823 13495 tablet.cc:2369] T 932fc63ec8e947f78e3faa8726d7489d P c6095c7c11104951be65e0b4c1c6c2f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 12:48:40.643450 13361 tablet.cc:2369] T 932fc63ec8e947f78e3faa8726d7489d P c3de391db8bc406383d179ec7ab6e5f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:48:40.656903 13671 raft_consensus.cc:482] T 932fc63ec8e947f78e3faa8726d7489d P c6095c7c11104951be65e0b4c1c6c2f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:48:40.657178 13671 raft_consensus.cc:504] T 932fc63ec8e947f78e3faa8726d7489d P c6095c7c11104951be65e0b4c1c6c2f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3de391db8bc406383d179ec7ab6e5f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32911 } } peers { permanent_uuid: "c6095c7c11104951be65e0b4c1c6c2f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46653 } } peers { permanent_uuid: "7d7f6633fc3645af8996eca5883e205e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43535 } }
I20260927 12:48:40.658425 13671 leader_election.cc:288] T 932fc63ec8e947f78e3faa8726d7489d P c6095c7c11104951be65e0b4c1c6c2f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3de391db8bc406383d179ec7ab6e5f5 (127.0.98.1:32911), 7d7f6633fc3645af8996eca5883e205e (127.0.98.3:43535)
W20260927 12:48:40.659642 13629 tablet.cc:2369] T 932fc63ec8e947f78e3faa8726d7489d P 7d7f6633fc3645af8996eca5883e205e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:48:40.677892 13657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:40.659302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50163 (local address 127.0.98.1:32911)
0927 12:48:40.659434 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 12:48:40.659439 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:48:40.664826 (+ 5387us) server_negotiation.cc:407] Connection header received
0927 12:48:40.665055 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:48:40.665059 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:48:40.665113 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:48:40.665160 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:48:40.665650 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:40.666474 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:40.676423 (+ 9949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:40.677061 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:40.677184 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:40.677267 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 12:48:40.677659 (+ 392us) server_negotiation.cc:299] Negotiation successful
0927 12:48:40.677755 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 12:48:40.678654 13672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:40.658843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32911 (local address 127.0.98.2:50163)
0927 12:48:40.664696 (+ 5853us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:40.664716 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 12:48:40.664923 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:48:40.665313 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:48:40.665316 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:48:40.665336 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:48:40.665485 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:40.665491 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:40.666637 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:40.666640 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:40.668011 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:40.668024 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:40.677338 (+ 9314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:40.677342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:40.677500 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:40.677527 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 12:48:40.678386 (+ 859us) client_negotiation.cc:240] Negotiation successful
0927 12:48:40.678479 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5745,"mutex_wait_us":29,"thread_start_us":127,"threads_started":1}
I20260927 12:48:40.679577 13314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "932fc63ec8e947f78e3faa8726d7489d" candidate_uuid: "c6095c7c11104951be65e0b4c1c6c2f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3de391db8bc406383d179ec7ab6e5f5" is_pre_election: true
I20260927 12:48:40.679940 13314 raft_consensus.cc:2454] T 932fc63ec8e947f78e3faa8726d7489d P c3de391db8bc406383d179ec7ab6e5f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6095c7c11104951be65e0b4c1c6c2f3 in term 0.
I20260927 12:48:40.680517 13383 leader_election.cc:302] T 932fc63ec8e947f78e3faa8726d7489d P c6095c7c11104951be65e0b4c1c6c2f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3de391db8bc406383d179ec7ab6e5f5, c6095c7c11104951be65e0b4c1c6c2f3; no voters:
I20260927 12:48:40.680897 13671 raft_consensus.cc:2781] T 932fc63ec8e947f78e3faa8726d7489d P c6095c7c11104951be65e0b4c1c6c2f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:48:40.680997 13671 raft_consensus.cc:482] T 932fc63ec8e947f78e3faa8726d7489d P c6095c7c11104951be65e0b4c1c6c2f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:48:40.681038 13671 raft_consensus.cc:3037] T 932fc63ec8e947f78e3faa8726d7489d P c6095c7c11104951be65e0b4c1c6c2f3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:48:40.682878 13671 raft_consensus.cc:504] T 932fc63ec8e947f78e3faa8726d7489d P c6095c7c11104951be65e0b4c1c6c2f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3de391db8bc406383d179ec7ab6e5f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32911 } } peers { permanent_uuid: "c6095c7c11104951be65e0b4c1c6c2f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46653 } } peers { permanent_uuid: "7d7f6633fc3645af8996eca5883e205e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43535 } }
I20260927 12:48:40.683315 13671 leader_election.cc:288] T 932fc63ec8e947f78e3faa8726d7489d P c6095c7c11104951be65e0b4c1c6c2f3 [CANDIDATE]: Term 1 election: Requested vote from peers c3de391db8bc406383d179ec7ab6e5f5 (127.0.98.1:32911), 7d7f6633fc3645af8996eca5883e205e (127.0.98.3:43535)
I20260927 12:48:40.684033 13314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "932fc63ec8e947f78e3faa8726d7489d" candidate_uuid: "c6095c7c11104951be65e0b4c1c6c2f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3de391db8bc406383d179ec7ab6e5f5"
I20260927 12:48:40.686702 13661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:40.660243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51329 (local address 127.0.98.3:43535)
0927 12:48:40.660354 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 12:48:40.660358 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:48:40.673051 (+ 12693us) server_negotiation.cc:407] Connection header received
0927 12:48:40.673283 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:48:40.673287 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:48:40.673343 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:48:40.673389 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:48:40.673834 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:40.674712 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:40.676443 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:40.677080 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:40.677171 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:40.677249 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 12:48:40.686440 (+ 9191us) server_negotiation.cc:299] Negotiation successful
0927 12:48:40.686561 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 12:48:40.687355 13673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:40.659853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43535 (local address 127.0.98.2:51329)
0927 12:48:40.672928 (+ 13075us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:40.672947 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 12:48:40.673153 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:48:40.673521 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:48:40.673524 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:48:40.673541 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:48:40.673663 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:40.673669 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:40.674865 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:40.674868 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:40.676215 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:40.676227 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:40.677360 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:40.677363 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:40.686211 (+ 8848us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:40.686248 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 12:48:40.687087 (+ 839us) client_negotiation.cc:240] Negotiation successful
0927 12:48:40.687186 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12971,"mutex_wait_us":8640,"thread_start_us":111,"threads_started":1}
I20260927 12:48:40.688395 13582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "932fc63ec8e947f78e3faa8726d7489d" candidate_uuid: "c6095c7c11104951be65e0b4c1c6c2f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d7f6633fc3645af8996eca5883e205e" is_pre_election: true
I20260927 12:48:40.688761 13582 raft_consensus.cc:2454] T 932fc63ec8e947f78e3faa8726d7489d P 7d7f6633fc3645af8996eca5883e205e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6095c7c11104951be65e0b4c1c6c2f3 in term 0.
I20260927 12:48:40.689085 13581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "932fc63ec8e947f78e3faa8726d7489d" candidate_uuid: "c6095c7c11104951be65e0b4c1c6c2f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d7f6633fc3645af8996eca5883e205e"
I20260927 12:48:40.689182 13581 raft_consensus.cc:3037] T 932fc63ec8e947f78e3faa8726d7489d P 7d7f6633fc3645af8996eca5883e205e [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:48:40.690613 13581 raft_consensus.cc:2454] T 932fc63ec8e947f78e3faa8726d7489d P 7d7f6633fc3645af8996eca5883e205e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6095c7c11104951be65e0b4c1c6c2f3 in term 1.
I20260927 12:48:40.691494 13384 leader_election.cc:302] T 932fc63ec8e947f78e3faa8726d7489d P c6095c7c11104951be65e0b4c1c6c2f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d7f6633fc3645af8996eca5883e205e, c6095c7c11104951be65e0b4c1c6c2f3; no voters:
I20260927 12:48:40.684175 13314 raft_consensus.cc:3037] T 932fc63ec8e947f78e3faa8726d7489d P c3de391db8bc406383d179ec7ab6e5f5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:48:40.693917 13314 raft_consensus.cc:2454] T 932fc63ec8e947f78e3faa8726d7489d P c3de391db8bc406383d179ec7ab6e5f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6095c7c11104951be65e0b4c1c6c2f3 in term 1.
I20260927 12:48:40.694607 13671 raft_consensus.cc:2781] T 932fc63ec8e947f78e3faa8726d7489d P c6095c7c11104951be65e0b4c1c6c2f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:48:40.694869 13671 raft_consensus.cc:686] T 932fc63ec8e947f78e3faa8726d7489d P c6095c7c11104951be65e0b4c1c6c2f3 [term 1 LEADER]: Becoming Leader. State: Replica: c6095c7c11104951be65e0b4c1c6c2f3, State: Running, Role: LEADER
I20260927 12:48:40.695273 13671 consensus_queue.cc:237] T 932fc63ec8e947f78e3faa8726d7489d P c6095c7c11104951be65e0b4c1c6c2f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3de391db8bc406383d179ec7ab6e5f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32911 } } peers { permanent_uuid: "c6095c7c11104951be65e0b4c1c6c2f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46653 } } peers { permanent_uuid: "7d7f6633fc3645af8996eca5883e205e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43535 } }
I20260927 12:48:40.699210 13156 catalog_manager.cc:5495] T 932fc63ec8e947f78e3faa8726d7489d P c6095c7c11104951be65e0b4c1c6c2f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6095c7c11104951be65e0b4c1c6c2f3 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "c6095c7c11104951be65e0b4c1c6c2f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3de391db8bc406383d179ec7ab6e5f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6095c7c11104951be65e0b4c1c6c2f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d7f6633fc3645af8996eca5883e205e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43535 } health_report { overall_health: UNKNOWN } } }
I20260927 12:48:40.748469 13659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:40.743651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52546 (local address 127.0.98.2:46653)
0927 12:48:40.744082 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0927 12:48:40.744087 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:48:40.744100 (+ 13us) server_negotiation.cc:407] Connection header received
0927 12:48:40.744188 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:48:40.744191 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:48:40.744275 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:48:40.744387 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:48:40.744710 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:40.745493 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:40.746819 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:40.747163 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:40.747258 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:40.747653 (+ 395us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:48:40.748086 (+ 433us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:48:40.748218 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 12:48:40.748249 (+ 31us) server_negotiation.cc:299] Negotiation successful
0927 12:48:40.748311 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345}
I20260927 12:48:40.754470 13581 raft_consensus.cc:1261] T 932fc63ec8e947f78e3faa8726d7489d P 7d7f6633fc3645af8996eca5883e205e [term 1 FOLLOWER]: Refusing update from remote peer c6095c7c11104951be65e0b4c1c6c2f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:48:40.754478 13314 raft_consensus.cc:1261] T 932fc63ec8e947f78e3faa8726d7489d P c3de391db8bc406383d179ec7ab6e5f5 [term 1 FOLLOWER]: Refusing update from remote peer c6095c7c11104951be65e0b4c1c6c2f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:48:40.755228 13674 consensus_queue.cc:1035] T 932fc63ec8e947f78e3faa8726d7489d P c6095c7c11104951be65e0b4c1c6c2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3de391db8bc406383d179ec7ab6e5f5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:48:40.755669 13671 consensus_queue.cc:1035] T 932fc63ec8e947f78e3faa8726d7489d P c6095c7c11104951be65e0b4c1c6c2f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d7f6633fc3645af8996eca5883e205e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:48:40.764281 13684 mvcc.cc:205] Tried to move back new op lower bound from 7333942561802395648 to 7333942561572024320. Current Snapshot: MvccSnapshot[applied={T|T < 7333942561802395648}]
I20260927 12:48:40.778198 13156 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:48:40.778379 13156 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:48:40.780761 13156 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 125e81691bcc4d5f860760aa8a45362c: Sending DeleteTablet for 3 replicas of tablet 932fc63ec8e947f78e3faa8726d7489d
I20260927 12:48:40.781582 13276 tablet_service.cc:1514] Processing DeleteTablet for tablet 932fc63ec8e947f78e3faa8726d7489d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:48:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58100
I20260927 12:48:40.781669 13562 tablet_service.cc:1514] Processing DeleteTablet for tablet 932fc63ec8e947f78e3faa8726d7489d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:48:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49632
I20260927 12:48:40.782493 13411 tablet_service.cc:1514] Processing DeleteTablet for tablet 932fc63ec8e947f78e3faa8726d7489d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:48:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52538
I20260927 12:48:40.782985 13696 tablet_replica.cc:330] T 932fc63ec8e947f78e3faa8726d7489d P c6095c7c11104951be65e0b4c1c6c2f3: stopping tablet replica
I20260927 12:48:40.783190 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 13200
I20260927 12:48:40.783355 13696 raft_consensus.cc:2229] T 932fc63ec8e947f78e3faa8726d7489d P c6095c7c11104951be65e0b4c1c6c2f3 [term 1 LEADER]: Raft consensus shutting down.
I20260927 12:48:40.784058 13696 raft_consensus.cc:2258] T 932fc63ec8e947f78e3faa8726d7489d P c6095c7c11104951be65e0b4c1c6c2f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:48:40.785910 13696 ts_tablet_manager.cc:1905] T 932fc63ec8e947f78e3faa8726d7489d P c6095c7c11104951be65e0b4c1c6c2f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:48:40.788645 13694 tablet_replica.cc:330] T 932fc63ec8e947f78e3faa8726d7489d P 7d7f6633fc3645af8996eca5883e205e: stopping tablet replica
I20260927 12:48:40.788753 13696 ts_tablet_manager.cc:1918] T 932fc63ec8e947f78e3faa8726d7489d P c6095c7c11104951be65e0b4c1c6c2f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:48:40.788950 13696 log.cc:1196] T 932fc63ec8e947f78e3faa8726d7489d P c6095c7c11104951be65e0b4c1c6c2f3: Deleting WAL directory at /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-1/wal/wals/932fc63ec8e947f78e3faa8726d7489d
I20260927 12:48:40.789049 13694 raft_consensus.cc:2229] T 932fc63ec8e947f78e3faa8726d7489d P 7d7f6633fc3645af8996eca5883e205e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:48:40.789410 13696 ts_tablet_manager.cc:1939] T 932fc63ec8e947f78e3faa8726d7489d P c6095c7c11104951be65e0b4c1c6c2f3: Deleting consensus metadata
I20260927 12:48:40.789433 13694 raft_consensus.cc:2258] T 932fc63ec8e947f78e3faa8726d7489d P 7d7f6633fc3645af8996eca5883e205e [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 12:48:40.789891 13148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:32911 (error 108)
I20260927 12:48:40.790848 13146 catalog_manager.cc:4841] TS c6095c7c11104951be65e0b4c1c6c2f3 (127.0.98.2:46653): tablet 932fc63ec8e947f78e3faa8726d7489d (table test-table [id=13e3becbd24f4154a3c8aaa970dbb26b]) successfully deleted
I20260927 12:48:40.792052 13658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:40.791673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32911 (local address 127.0.0.1:58112)
0927 12:48:40.791891 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:40.791962 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
W20260927 12:48:40.792194 13148 catalog_manager.cc:4584] TS c3de391db8bc406383d179ec7ab6e5f5 (127.0.98.1:32911): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 932fc63ec8e947f78e3faa8726d7489d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32911: connect: Connection refused (error 111)
I20260927 12:48:40.792397 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 13363
I20260927 12:48:40.793838 13694 ts_tablet_manager.cc:1905] T 932fc63ec8e947f78e3faa8726d7489d P 7d7f6633fc3645af8996eca5883e205e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:48:40.796131 13694 ts_tablet_manager.cc:1918] T 932fc63ec8e947f78e3faa8726d7489d P 7d7f6633fc3645af8996eca5883e205e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:48:40.796241 13694 log.cc:1196] T 932fc63ec8e947f78e3faa8726d7489d P 7d7f6633fc3645af8996eca5883e205e: Deleting WAL directory at /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790513161160093-392-0/minicluster-data/ts-2/wal/wals/932fc63ec8e947f78e3faa8726d7489d
I20260927 12:48:40.796530 13694 ts_tablet_manager.cc:1939] T 932fc63ec8e947f78e3faa8726d7489d P 7d7f6633fc3645af8996eca5883e205e: Deleting consensus metadata
I20260927 12:48:40.797778 13149 catalog_manager.cc:4841] TS 7d7f6633fc3645af8996eca5883e205e (127.0.98.3:43535): tablet 932fc63ec8e947f78e3faa8726d7489d (table test-table [id=13e3becbd24f4154a3c8aaa970dbb26b]) successfully deleted
I20260927 12:48:40.799096 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 13497
I20260927 12:48:40.805393 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 13129
I20260927 12:48:40.812661 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 12:48:41.147378 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 12:48:41.147487 392 mini_ranger.cc:66] Stopping Ranger...
I20260927 12:48:41.482250 392 mini_ranger.cc:68] Stopped Ranger
2026-09-27 12:48:41.482 UTC [12196] LOG: received smart shutdown request
2026-09-27 12:48:41.486 UTC [12196] LOG: background worker "logical replication launcher" (PID 12205) exited with exit code 1
2026-09-27 12:48:41.486 UTC [12200] LOG: shutting down
2026-09-27 12:48:41.671 UTC [12196] LOG: database system is shut down
2026-09-27T12:48:41Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48028 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:48:41 dist-test-slave-lkn4 krb5kdc[13706](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:48:41 dist-test-slave-lkn4 krb5kdc[13706](info): set up 2 sockets
Sep 27 12:48:41 dist-test-slave-lkn4 krb5kdc[13706](info): commencing operation
krb5kdc: starting...
W20260927 12:48:42.742118 392 mini_kdc.cc:121] Time spent starting KDC: real 0.951s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/krb5kdc/test-user.keytab.
Sep 27 12:48:42 dist-test-slave-lkn4 krb5kdc[13706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:48:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:48:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:48:42.862962 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32849
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32849,127.0.98.61:46363,127.0.98.60:37657
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:48:42.971504 13719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:48:42.971794 13719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:48:42.971863 13719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:48:42.975953 13719 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:48:42.976025 13719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:48:42.976052 13719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:48:42.976073 13719 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:48:42.976094 13719 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:48:42.981429 13719 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32849,127.0.98.61:46363,127.0.98.60:37657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:48:42.982540 13719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:48:42.983553 13719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:48:42.991918 13727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:48:42.991946 13725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:48:42.992002 13719 server_base.cc:1034] running on GCE node
W20260927 12:48:42.991946 13724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:48:42.992679 13719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:48:42.993749 13719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:48:42.994923 13719 hybrid_clock.cc:648] HybridClock initialized: now 1790513322994886 us; error 48 us; skew 500 ppm
Sep 27 12:48:42 dist-test-slave-lkn4 krb5kdc[13706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513322, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:48:42.997694 13719 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:48:42.998589 13719 webserver.cc:460] Webserver started at http://127.0.98.62:41879/ using document root <none> and password file <none>
I20260927 12:48:42.999109 13719 fs_manager.cc:359] Metadata directory not provided
I20260927 12:48:42.999191 13719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:48:42.999390 13719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:48:43.001381 13719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/minicluster-data/master-0/data/instance:
uuid: "5b31d259044b44cbb67469066512838c"
format_stamp: "Formatted at 2026-09-27 12:48:42 on dist-test-slave-lkn4"
server_key: "93aa238066c84182d6839c5e3b0b24ca"
server_key_iv: "0822772b5c7005737c983e0186cf9582"
server_key_version: "encryptionkey@0"
I20260927 12:48:43.001899 13719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/minicluster-data/master-0/wal/instance:
uuid: "5b31d259044b44cbb67469066512838c"
format_stamp: "Formatted at 2026-09-27 12:48:42 on dist-test-slave-lkn4"
server_key: "93aa238066c84182d6839c5e3b0b24ca"
server_key_iv: "0822772b5c7005737c983e0186cf9582"
server_key_version: "encryptionkey@0"
I20260927 12:48:43.005465 13719 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 12:48:43.007836 13734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:48:43.009146 13719 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 12:48:43.009277 13719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/minicluster-data/master-0/wal
uuid: "5b31d259044b44cbb67469066512838c"
format_stamp: "Formatted at 2026-09-27 12:48:42 on dist-test-slave-lkn4"
server_key: "93aa238066c84182d6839c5e3b0b24ca"
server_key_iv: "0822772b5c7005737c983e0186cf9582"
server_key_version: "encryptionkey@0"
I20260927 12:48:43.009387 13719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:48:43.024011 13719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:48:43.027046 13719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:48:43.027218 13719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:48:43.033733 13719 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32849
I20260927 12:48:43.033773 13786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32849 every 8 connection(s)
I20260927 12:48:43.034727 13719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:48:43.037377 13787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:48:43.038945 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 13719
I20260927 12:48:43.039065 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/minicluster-data/master-0/wal/instance
I20260927 12:48:43.039354 392 external_mini_cluster.cc:1428] Setting key b98009aa4ce26ba8fca9b67411210ee0
I20260927 12:48:43.041793 13787 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32849 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 27 12:48:43 dist-test-slave-lkn4 krb5kdc[13706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513322, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:48:43.057183 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:43.044206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32849 (local address 127.0.0.1:59378)
0927 12:48:43.044842 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:43.044880 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 12:48:43.045680 (+ 800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:48:43.046372 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:48:43.046378 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:48:43.046812 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:48:43.047089 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:43.047104 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:43.048299 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:43.048301 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:43.049037 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:43.049046 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:43.049758 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:43.049766 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:43.049875 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:43.050465 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:48:43.050482 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:48:43.051919 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:48:43.054217 (+ 2298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:48:43.054225 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:48:43.054234 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:48:43.054478 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:48:43.054846 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:48:43.054848 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:48:43.054849 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:48:43.054958 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:48:43.055705 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:48:43.055710 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:48:43.055836 (+ 126us) client_negotiation.cc:769] Sending connection context
0927 12:48:43.055974 (+ 138us) client_negotiation.cc:240] Negotiation successful
0927 12:48:43.056315 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"mutex_wait_us":59,"spinlock_wait_cycles":6400,"thread_start_us":139,"threads_started":1}
I20260927 12:48:43.057186 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:43.044196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59378 (local address 127.0.98.62:32849)
0927 12:48:43.044617 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0927 12:48:43.044635 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 12:48:43.044980 (+ 345us) server_negotiation.cc:407] Connection header received
0927 12:48:43.045969 (+ 989us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:48:43.045974 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:48:43.046197 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:48:43.046364 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:48:43.047230 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:43.048179 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:43.049150 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:43.049512 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:43.049657 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:43.052075 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:48:43.052081 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:48:43.052096 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:48:43.052120 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:48:43.054097 (+ 1977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:48:43.054586 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:48:43.054593 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:48:43.054595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:48:43.054749 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:48:43.055042 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:48:43.055045 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:48:43.055047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:48:43.055595 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:48:43.055747 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 12:48:43.056075 (+ 328us) server_negotiation.cc:299] Negotiation successful
0927 12:48:43.056399 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":96,"server-negotiator.queue_time_us":281,"thread_start_us":118,"threads_started":1}
I20260927 12:48:43.059646 13787 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46363 } has no permanent_uuid. Determining permanent_uuid...
I20260927 12:48:43.060388 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:43.060122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46363 (local address 127.0.0.1:38556)
0927 12:48:43.060229 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:43.060316 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 12:48:43.060572 13736 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46363: connect: Connection refused (error 111)
I20260927 12:48:43.061466 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:43.061268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46363 (local address 127.0.0.1:38564)
0927 12:48:43.061365 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:43.061411 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260927 12:48:43.061600 13787 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46363: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46363: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 12:48:43.065851 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:46363
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32849,127.0.98.61:46363,127.0.98.60:37657
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 12:48:43.110857 13787 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46363 } attempt: 1
I20260927 12:48:43.111754 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:43.111446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46363 (local address 127.0.0.1:38574)
0927 12:48:43.111600 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:43.111665 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 12:48:43.112484 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:43.112272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46363 (local address 127.0.0.1:38580)
0927 12:48:43.112379 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:43.112424 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 12:48:43.112597 13787 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46363: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46363: connect: Connection refused (error 111)
W20260927 12:48:43.175561 13794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:48:43.175866 13794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:48:43.175909 13794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:48:43.179996 13794 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:48:43.180060 13794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:48:43.180078 13794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:48:43.180126 13794 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:48:43.180143 13794 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:48:43.180732 13787 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46363 } attempt: 2
I20260927 12:48:43.181545 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:43.181253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46363 (local address 127.0.0.1:38594)
0927 12:48:43.181406 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:43.181469 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 12:48:43.182224 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:43.182017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46363 (local address 127.0.0.1:38598)
0927 12:48:43.182120 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:43.182165 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 12:48:43.182331 13787 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46363: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46363: connect: Connection refused (error 111)
I20260927 12:48:43.185845 13794 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32849,127.0.98.61:46363,127.0.98.60:37657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:46363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:48:43.186936 13794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:48:43.187975 13794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:48:43.196195 13800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:48:43.196205 13799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:48:43.196271 13794 server_base.cc:1034] running on GCE node
W20260927 12:48:43.196208 13802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:48:43.197007 13794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:48:43.198130 13794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:48:43.199286 13794 hybrid_clock.cc:648] HybridClock initialized: now 1790513323199270 us; error 34 us; skew 500 ppm
Sep 27 12:48:43 dist-test-slave-lkn4 krb5kdc[13706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513323, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:48:43.201982 13794 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 12:48:43.202875 13794 webserver.cc:460] Webserver started at http://127.0.98.61:42917/ using document root <none> and password file <none>
I20260927 12:48:43.203379 13794 fs_manager.cc:359] Metadata directory not provided
I20260927 12:48:43.203433 13794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:48:43.203601 13794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:48:43.205595 13794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/minicluster-data/master-1/data/instance:
uuid: "090d21945ebe484f96e97f35a88ef28e"
format_stamp: "Formatted at 2026-09-27 12:48:43 on dist-test-slave-lkn4"
server_key: "237da55ae65d5128878e784878119e2f"
server_key_iv: "e8629a29f6c2688fbad08a04f151ede1"
server_key_version: "encryptionkey@0"
I20260927 12:48:43.206151 13794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/minicluster-data/master-1/wal/instance:
uuid: "090d21945ebe484f96e97f35a88ef28e"
format_stamp: "Formatted at 2026-09-27 12:48:43 on dist-test-slave-lkn4"
server_key: "237da55ae65d5128878e784878119e2f"
server_key_iv: "e8629a29f6c2688fbad08a04f151ede1"
server_key_version: "encryptionkey@0"
I20260927 12:48:43.209746 13794 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260927 12:48:43.212248 13809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:48:43.213578 13794 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260927 12:48:43.213685 13794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/minicluster-data/master-1/wal
uuid: "090d21945ebe484f96e97f35a88ef28e"
format_stamp: "Formatted at 2026-09-27 12:48:43 on dist-test-slave-lkn4"
server_key: "237da55ae65d5128878e784878119e2f"
server_key_iv: "e8629a29f6c2688fbad08a04f151ede1"
server_key_version: "encryptionkey@0"
I20260927 12:48:43.213811 13794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:48:43.230736 13794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:48:43.233963 13794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:48:43.234169 13794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:48:43.240815 13794 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:46363
I20260927 12:48:43.240828 13861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:46363 every 8 connection(s)
I20260927 12:48:43.241876 13794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/minicluster-data/master-1/data/info.pb
I20260927 12:48:43.244735 13862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:48:43.248600 13862 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32849 } has no permanent_uuid. Determining permanent_uuid...
I20260927 12:48:43.251745 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 13794
I20260927 12:48:43.251879 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/minicluster-data/master-1/wal/instance
I20260927 12:48:43.252157 392 external_mini_cluster.cc:1428] Setting key 09578f70cc777b02ada45262523bb405
Sep 27 12:48:43 dist-test-slave-lkn4 krb5kdc[13706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513323, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260927 12:48:43.262367 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:43.250575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59392 (local address 127.0.98.62:32849)
0927 12:48:43.250714 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 12:48:43.250717 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:48:43.251602 (+ 885us) server_negotiation.cc:407] Connection header received
0927 12:48:43.252784 (+ 1182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:48:43.252787 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:48:43.252837 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:48:43.252916 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:48:43.254343 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:43.255059 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:43.256189 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:43.256436 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:43.256529 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:43.259126 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:48:43.259129 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:48:43.259131 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:48:43.259156 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:48:43.260449 (+ 1293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:48:43.260906 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:48:43.260910 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:48:43.260912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:48:43.260984 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:48:43.261289 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:48:43.261292 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:48:43.261294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:48:43.261471 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:48:43.261553 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 12:48:43.262079 (+ 526us) server_negotiation.cc:299] Negotiation successful
0927 12:48:43.262205 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260927 12:48:43.263219 13864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:43.250952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32849 (local address 127.0.0.1:59392)
0927 12:48:43.251428 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:43.251477 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 12:48:43.252532 (+ 1055us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:48:43.253116 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:48:43.253125 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:48:43.253582 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:48:43.254201 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:43.254216 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:43.255172 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:43.255175 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:43.256065 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:43.256073 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:43.256584 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:43.256587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:43.256745 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:43.257372 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:48:43.257388 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:48:43.258968 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:48:43.260553 (+ 1585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:48:43.260559 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:48:43.260575 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:48:43.260811 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:48:43.261061 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:48:43.261063 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:48:43.261065 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:48:43.261207 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:48:43.261565 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:48:43.261571 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:48:43.261824 (+ 253us) client_negotiation.cc:769] Sending connection context
0927 12:48:43.262006 (+ 182us) client_negotiation.cc:240] Negotiation successful
0927 12:48:43.262361 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":122,"threads_started":1}
I20260927 12:48:43.264693 13862 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46363 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 12:48:43 dist-test-slave-lkn4 krb5kdc[13706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513323, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 12:48:43.273479 13787 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46363 } attempt: 3
I20260927 12:48:43.273820 13864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:43.265206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46363 (local address 127.0.0.1:38614)
0927 12:48:43.265310 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:43.265324 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:48:43.265400 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:48:43.265899 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:48:43.265903 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:48:43.266092 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:48:43.266205 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:43.266211 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:43.267286 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:43.267290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:43.267823 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:43.267831 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:43.268655 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:43.268658 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:43.268817 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:43.269279 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:48:43.269290 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:48:43.270467 (+ 1177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:48:43.272233 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:48:43.272236 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:48:43.272237 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:48:43.272442 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:48:43.272810 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:48:43.272813 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:48:43.272815 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:48:43.272889 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:48:43.273388 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:48:43.273390 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:48:43.273490 (+ 100us) client_negotiation.cc:769] Sending connection context
0927 12:48:43.273569 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 12:48:43.273687 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260927 12:48:43.274474 13867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:43.265278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38614 (local address 127.0.98.61:46363)
0927 12:48:43.265552 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0927 12:48:43.265560 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 12:48:43.265578 (+ 18us) server_negotiation.cc:407] Connection header received
0927 12:48:43.265614 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:48:43.265620 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:48:43.265782 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:48:43.265882 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:48:43.266298 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:43.267185 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:43.267927 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:43.268512 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:43.268618 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:43.270593 (+ 1975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:48:43.270602 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:48:43.270618 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:48:43.270652 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:48:43.272133 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:48:43.272523 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:48:43.272528 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:48:43.272529 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:48:43.272595 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:48:43.273003 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:48:43.273007 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:48:43.273008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:48:43.273293 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:48:43.273431 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 12:48:43.273695 (+ 264us) server_negotiation.cc:299] Negotiation successful
0927 12:48:43.273857 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":129,"threads_started":1}
I20260927 12:48:43.276361 13862 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37657 } has no permanent_uuid. Determining permanent_uuid...
I20260927 12:48:43.277045 13864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:43.276763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37657 (local address 127.0.0.1:45728)
0927 12:48:43.276905 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:43.276981 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 12:48:43.277576 13811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37657: connect: Connection refused (error 111)
I20260927 12:48:43.278486 13864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:43.278235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37657 (local address 127.0.0.1:45738)
0927 12:48:43.278339 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:43.278411 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/krb5kdc/kudu_127.0.98.60.keytab.
W20260927 12:48:43.278705 13862 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37657: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37657: connect: Connection refused (error 111)
Sep 27 12:48:43 dist-test-slave-lkn4 krb5kdc[13706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513322, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 12:48:43.279302 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37657
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32849,127.0.98.61:46363,127.0.98.60:37657
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 12:48:43.282513 13868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:43.274104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38616 (local address 127.0.98.61:46363)
0927 12:48:43.274443 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0927 12:48:43.274453 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 12:48:43.274470 (+ 17us) server_negotiation.cc:407] Connection header received
0927 12:48:43.274507 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:48:43.274511 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:48:43.274573 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:48:43.274682 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:48:43.275063 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:43.275706 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:43.277259 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:43.277545 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:43.277637 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:43.279769 (+ 2132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:48:43.279773 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:48:43.279776 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:48:43.279799 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:48:43.281092 (+ 1293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:48:43.281523 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:48:43.281526 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:48:43.281527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:48:43.281591 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:48:43.281833 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:48:43.281835 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:48:43.281836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:48:43.281982 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:48:43.282083 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 12:48:43.282317 (+ 234us) server_negotiation.cc:299] Negotiation successful
0927 12:48:43.282403 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":108,"threads_started":1}
I20260927 12:48:43.282552 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:43.274115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46363 (local address 127.0.0.1:38616)
0927 12:48:43.274232 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:43.274245 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:48:43.274326 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:48:43.274704 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:48:43.274707 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:48:43.274856 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:48:43.274962 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:43.274968 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:43.275854 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:43.275858 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:43.277151 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:43.277163 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:43.277659 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:43.277663 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:43.277861 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:43.278344 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:48:43.278358 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:48:43.279617 (+ 1259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:48:43.281193 (+ 1576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:48:43.281196 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:48:43.281198 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:48:43.281442 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:48:43.281665 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:48:43.281667 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:48:43.281669 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:48:43.281754 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:48:43.282077 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:48:43.282080 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:48:43.282176 (+ 96us) client_negotiation.cc:769] Sending connection context
0927 12:48:43.282277 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 12:48:43.282407 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260927 12:48:43.283313 13787 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37657 } has no permanent_uuid. Determining permanent_uuid...
I20260927 12:48:43.283995 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:43.283742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37657 (local address 127.0.0.1:45742)
0927 12:48:43.283867 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:43.283923 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 12:48:43.284634 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:43.284378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37657 (local address 127.0.0.1:45758)
0927 12:48:43.284520 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:43.284570 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260927 12:48:43.284747 13787 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37657: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37657: connect: Connection refused (error 111)
I20260927 12:48:43.315913 13787 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37657 } attempt: 1
I20260927 12:48:43.316776 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:43.316433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37657 (local address 127.0.0.1:45770)
0927 12:48:43.316601 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:43.316678 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260927 12:48:43.317451 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:43.317246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37657 (local address 127.0.0.1:45776)
0927 12:48:43.317341 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:43.317392 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260927 12:48:43.317580 13787 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37657: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37657: connect: Connection refused (error 111)
I20260927 12:48:43.327999 13862 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37657 } attempt: 1
I20260927 12:48:43.328864 13864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:43.328545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37657 (local address 127.0.0.1:45780)
0927 12:48:43.328680 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:43.328743 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 12:48:43.329721 13864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:43.329403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37657 (local address 127.0.0.1:45782)
0927 12:48:43.329528 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:43.329580 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 12:48:43.329867 13862 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37657: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37657: connect: Connection refused (error 111)
W20260927 12:48:43.390576 13870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:48:43.390867 13870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:48:43.390908 13870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 12:48:43.392761 13787 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37657 } attempt: 2
I20260927 12:48:43.393564 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:43.393244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37657 (local address 127.0.0.1:45786)
0927 12:48:43.393400 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:43.393469 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 12:48:43.394227 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:43.393997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37657 (local address 127.0.0.1:45800)
0927 12:48:43.394116 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:43.394166 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260927 12:48:43.394347 13787 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37657: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37657: connect: Connection refused (error 111)
W20260927 12:48:43.395304 13870 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:48:43.395354 13870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:48:43.395371 13870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:48:43.395387 13870 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:48:43.395408 13870 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:48:43.398075 13862 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37657 } attempt: 2
I20260927 12:48:43.398799 13864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:43.398535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37657 (local address 127.0.0.1:45806)
0927 12:48:43.398665 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:43.398723 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 12:48:43.399430 13864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:43.399214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37657 (local address 127.0.0.1:45810)
0927 12:48:43.399325 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:43.399373 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 12:48:43.399534 13862 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37657: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37657: connect: Connection refused (error 111)
I20260927 12:48:43.401170 13870 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32849,127.0.98.61:46363,127.0.98.60:37657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:48:43.402313 13870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:48:43.403319 13870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:48:43.411808 13875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:48:43.411867 13876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:48:43.411881 13878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:48:43.412061 13870 server_base.cc:1034] running on GCE node
I20260927 12:48:43.412684 13870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:48:43.413854 13870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:48:43.415017 13870 hybrid_clock.cc:648] HybridClock initialized: now 1790513323414975 us; error 58 us; skew 500 ppm
Sep 27 12:48:43 dist-test-slave-lkn4 krb5kdc[13706](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513323, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:48:43.417927 13870 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 12:48:43.418881 13870 webserver.cc:460] Webserver started at http://127.0.98.60:42177/ using document root <none> and password file <none>
I20260927 12:48:43.419422 13870 fs_manager.cc:359] Metadata directory not provided
I20260927 12:48:43.419480 13870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:48:43.419647 13870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:48:43.421646 13870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/minicluster-data/master-2/data/instance:
uuid: "6fa0998ef5f541719a1af35b4c2c3b41"
format_stamp: "Formatted at 2026-09-27 12:48:43 on dist-test-slave-lkn4"
server_key: "bd22715df90ed309023e073fc510e6c4"
server_key_iv: "9b0aebe6c1c56db83a403c81000daa3f"
server_key_version: "encryptionkey@0"
I20260927 12:48:43.422190 13870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/minicluster-data/master-2/wal/instance:
uuid: "6fa0998ef5f541719a1af35b4c2c3b41"
format_stamp: "Formatted at 2026-09-27 12:48:43 on dist-test-slave-lkn4"
server_key: "bd22715df90ed309023e073fc510e6c4"
server_key_iv: "9b0aebe6c1c56db83a403c81000daa3f"
server_key_version: "encryptionkey@0"
I20260927 12:48:43.425774 13870 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:48:43.428207 13885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:48:43.429522 13870 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:48:43.429623 13870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/minicluster-data/master-2/wal
uuid: "6fa0998ef5f541719a1af35b4c2c3b41"
format_stamp: "Formatted at 2026-09-27 12:48:43 on dist-test-slave-lkn4"
server_key: "bd22715df90ed309023e073fc510e6c4"
server_key_iv: "9b0aebe6c1c56db83a403c81000daa3f"
server_key_version: "encryptionkey@0"
I20260927 12:48:43.429730 13870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:48:43.451190 13870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:48:43.454380 13870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:48:43.454576 13870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:48:43.461817 13870 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:37657
I20260927 12:48:43.461867 13937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:37657 every 8 connection(s)
I20260927 12:48:43.462801 13870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/minicluster-data/master-2/data/info.pb
I20260927 12:48:43.465684 13938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:48:43.465687 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 13870
I20260927 12:48:43.465793 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790513161160093-392-0/minicluster-data/master-2/wal/instance
I20260927 12:48:43.466099 392 external_mini_cluster.cc:1428] Setting key 97085b77d324f92328142d15ef3accee
I20260927 12:48:43.470793 13938 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32849 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 12:48:43 dist-test-slave-lkn4 krb5kdc[13706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 12:48:43 dist-test-slave-lkn4 krb5kdc[13706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513323, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:48:43.480551 13941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:43.467359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45824 (local address 127.0.98.60:37657)
0927 12:48:43.467725 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0927 12:48:43.467741 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 12:48:43.467776 (+ 35us) server_negotiation.cc:407] Connection header received
0927 12:48:43.468444 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:48:43.468463 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:48:43.468795 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:48:43.469102 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:48:43.470098 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:43.471065 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:43.471912 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:43.472420 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:43.472567 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:43.475149 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:48:43.475156 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:48:43.475167 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:48:43.475197 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:48:43.477218 (+ 2021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:48:43.477754 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:48:43.477760 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:48:43.477764 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:48:43.477887 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:48:43.478204 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:48:43.478207 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:48:43.478208 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:48:43.478650 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:48:43.478873 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0927 12:48:43.479225 (+ 352us) server_negotiation.cc:299] Negotiation successful
0927 12:48:43.479501 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":93,"threads_started":1}
I20260927 12:48:43.482064 13942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:43.472441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32849 (local address 127.0.0.1:59400)
0927 12:48:43.472961 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:43.473019 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 12:48:43.473191 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:48:43.473497 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:48:43.473502 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:48:43.473902 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:48:43.474096 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:43.474104 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:43.474978 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:43.474981 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:43.475889 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:43.475900 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:43.476349 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:43.476353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:43.476462 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:43.476876 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:48:43.476891 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:48:43.478407 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:48:43.480305 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:48:43.480313 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:48:43.480318 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:48:43.480539 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:48:43.481017 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:48:43.481020 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:48:43.481022 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:48:43.481095 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:48:43.481479 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:48:43.481485 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:48:43.481587 (+ 102us) client_negotiation.cc:769] Sending connection context
0927 12:48:43.481702 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 12:48:43.481875 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":160,"threads_started":1}
I20260927 12:48:43.482118 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:43.472545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59400 (local address 127.0.98.62:32849)
0927 12:48:43.472697 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 12:48:43.472702 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:48:43.473100 (+ 398us) server_negotiation.cc:407] Connection header received
0927 12:48:43.473305 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:48:43.473308 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:48:43.473361 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:48:43.473449 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:48:43.474232 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:43.474844 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:43.476016 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:43.476230 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:43.476315 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:43.478542 (+ 2227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:48:43.478546 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:48:43.478548 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:48:43.478577 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:48:43.480184 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:48:43.480831 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:48:43.480834 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:48:43.480836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:48:43.480924 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:48:43.481177 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:48:43.481180 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:48:43.481182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:48:43.481374 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:48:43.481463 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 12:48:43.481840 (+ 377us) server_negotiation.cc:299] Negotiation successful
0927 12:48:43.481954 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 12:48:43.483166 13938 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46363 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 12:48:43 dist-test-slave-lkn4 krb5kdc[13706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513323, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 27 12:48:43 dist-test-slave-lkn4 krb5kdc[13706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:48:43.490696 13862 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37657 } attempt: 3
I20260927 12:48:43.491051 13942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:43.483646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46363 (local address 127.0.0.1:38618)
0927 12:48:43.483797 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:43.483811 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:48:43.483878 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:48:43.484092 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:48:43.484095 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:48:43.484225 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:48:43.484318 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:43.484323 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:43.485140 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:43.485144 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:43.485883 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:43.485891 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:43.486395 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:43.486399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:43.486492 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:43.486844 (+ 352us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:48:43.486857 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:48:43.488018 (+ 1161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:48:43.489590 (+ 1572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:48:43.489593 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:48:43.489595 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:48:43.489772 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:48:43.490087 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:48:43.490090 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:48:43.490092 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:48:43.490175 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:48:43.490548 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:48:43.490550 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:48:43.490637 (+ 87us) client_negotiation.cc:769] Sending connection context
0927 12:48:43.490742 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 12:48:43.490890 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20260927 12:48:43.491442 13868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:43.483733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38618 (local address 127.0.98.61:46363)
0927 12:48:43.483838 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 12:48:43.483842 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:48:43.483856 (+ 14us) server_negotiation.cc:407] Connection header received
0927 12:48:43.483957 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:48:43.483959 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:48:43.483995 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:48:43.484067 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:48:43.484410 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:43.485029 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:43.485979 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:43.486285 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:43.486370 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:43.488143 (+ 1773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:48:43.488146 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:48:43.488148 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:48:43.488171 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:48:43.489492 (+ 1321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:48:43.489889 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:48:43.489893 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:48:43.489895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:48:43.489969 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:48:43.490253 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:48:43.490256 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:48:43.490257 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:48:43.490459 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:48:43.490554 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 12:48:43.490892 (+ 338us) server_negotiation.cc:299] Negotiation successful
0927 12:48:43.491021 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 12:48:43.491801 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:43.482879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59412 (local address 127.0.98.62:32849)
0927 12:48:43.482991 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 12:48:43.482995 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:48:43.483092 (+ 97us) server_negotiation.cc:407] Connection header received
0927 12:48:43.483194 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:48:43.483197 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:48:43.483240 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:48:43.483307 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:48:43.484072 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:43.484701 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:43.485694 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:43.485933 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:43.486026 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:43.488724 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:48:43.488727 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:48:43.488729 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:48:43.488751 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:48:43.490086 (+ 1335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:48:43.490477 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:48:43.490480 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:48:43.490482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:48:43.490563 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:48:43.490831 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:48:43.490834 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:48:43.490836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:48:43.491041 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:48:43.491242 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0927 12:48:43.491548 (+ 306us) server_negotiation.cc:299] Negotiation successful
0927 12:48:43.491659 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 12:48:43.492292 13938 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37657 } has no permanent_uuid. Determining permanent_uuid...
I20260927 12:48:43.493470 13787 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37657 } attempt: 3
Sep 27 12:48:43 dist-test-slave-lkn4 krb5kdc[13706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513323, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 12:48:43 dist-test-slave-lkn4 krb5kdc[13706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513322, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 12:48:43 dist-test-slave-lkn4 krb5kdc[13706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513323, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 12:48:43 dist-test-slave-lkn4 krb5kdc[13706](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 12:48:43.502553 13864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:43.491131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37657 (local address 127.0.0.1:45830)
0927 12:48:43.491495 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:43.491510 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:48:43.491582 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:48:43.492269 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:48:43.492272 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:48:43.492446 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:48:43.492550 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:43.492556 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:43.493547 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:43.493551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:43.494356 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:43.494366 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:43.494929 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:43.494932 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:43.495016 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:43.495508 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:48:43.495522 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:48:43.496972 (+ 1450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:48:43.499428 (+ 2456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:48:43.499431 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:48:43.499433 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:48:43.499798 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:48:43.500209 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:48:43.500213 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:48:43.500215 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:48:43.500303 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:48:43.501671 (+ 1368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:48:43.501675 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:48:43.502073 (+ 398us) client_negotiation.cc:769] Sending connection context
0927 12:48:43.502196 (+ 123us) client_negotiation.cc:240] Negotiation successful
0927 12:48:43.502350 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":3}
I20260927 12:48:43.502858 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:43.493796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37657 (local address 127.0.0.1:45840)
0927 12:48:43.493999 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:43.494011 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:48:43.494112 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:48:43.494669 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:48:43.494671 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:48:43.494839 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:48:43.494938 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:43.494944 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:43.496009 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:43.496013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:43.496801 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:43.496812 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:43.497410 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:43.497413 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:43.497494 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:43.497944 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:48:43.497959 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:48:43.499441 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:48:43.501318 (+ 1877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:48:43.501322 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:48:43.501324 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:48:43.501553 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:48:43.501842 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:48:43.501845 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:48:43.501848 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:48:43.501934 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:48:43.502393 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:48:43.502397 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:48:43.502498 (+ 101us) client_negotiation.cc:769] Sending connection context
0927 12:48:43.502576 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 12:48:43.502702 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139}
I20260927 12:48:43.503531 13944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:43.494225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45840 (local address 127.0.98.60:37657)
0927 12:48:43.494479 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0927 12:48:43.494487 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 12:48:43.494497 (+ 10us) server_negotiation.cc:407] Connection header received
0927 12:48:43.494520 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:48:43.494522 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:48:43.494556 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:48:43.495031 (+ 475us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:48:43.495142 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:43.495889 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:43.496940 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:43.497272 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:43.497666 (+ 394us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:43.499582 (+ 1916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:48:43.499586 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:48:43.499588 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:48:43.499613 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:48:43.501201 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:48:43.501676 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:48:43.501679 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:48:43.501680 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:48:43.501759 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:48:43.502034 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:48:43.502062 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:48:43.502064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:48:43.502268 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:48:43.503155 (+ 887us) server_negotiation.cc:1035] Waiting for connection context
0927 12:48:43.503226 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 12:48:43.503357 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":201,"thread_start_us":107,"threads_started":1}
I20260927 12:48:43.504009 13941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:43.491725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45830 (local address 127.0.98.60:37657)
0927 12:48:43.491830 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 12:48:43.491834 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:48:43.491845 (+ 11us) server_negotiation.cc:407] Connection header received
0927 12:48:43.491875 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:48:43.492097 (+ 222us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:48:43.492147 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:48:43.492650 (+ 503us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:48:43.492736 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:43.493436 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:43.494484 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:43.494811 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:43.497077 (+ 2266us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:43.497169 (+ 92us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:48:43.497173 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:48:43.497175 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:48:43.497202 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:48:43.499302 (+ 2100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:48:43.499929 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:48:43.499933 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:48:43.499935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:48:43.500037 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:48:43.500640 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:48:43.500643 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:48:43.500645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:48:43.501169 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:48:43.503369 (+ 2200us) server_negotiation.cc:1035] Waiting for connection context
0927 12:48:43.503439 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 12:48:43.503557 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32,"spinlock_wait_cycles":1408}
I20260927 12:48:43.504338 13942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:43.492880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37657 (local address 127.0.0.1:45832)
0927 12:48:43.493280 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:43.493304 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 12:48:43.493366 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:48:43.493902 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:48:43.493905 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:48:43.494089 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:48:43.494182 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:43.494191 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:43.495693 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:43.495697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:43.496457 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:43.496473 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:43.497256 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:43.497260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:43.497338 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:43.497798 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:48:43.497815 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:48:43.500455 (+ 2640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:48:43.502458 (+ 2003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:48:43.502461 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:48:43.502463 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:48:43.502702 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:48:43.503001 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:48:43.503003 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:48:43.503005 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:48:43.503081 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:48:43.503481 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:48:43.503484 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:48:43.503578 (+ 94us) client_negotiation.cc:769] Sending connection context
0927 12:48:43.503696 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 12:48:43.503815 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"mutex_wait_us":1}
I20260927 12:48:43.504472 13868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:43.493143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38620 (local address 127.0.98.61:46363)
0927 12:48:43.493251 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 12:48:43.493256 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:48:43.493341 (+ 85us) server_negotiation.cc:407] Connection header received
0927 12:48:43.493548 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:48:43.493552 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:48:43.493605 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:48:43.493663 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:48:43.494656 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:43.495423 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:43.496554 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:43.496880 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:43.496959 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:43.501062 (+ 4103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:48:43.501066 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:48:43.501069 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:48:43.501094 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:48:43.502611 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:48:43.503073 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:48:43.503077 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:48:43.503079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:48:43.503160 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:48:43.503459 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:48:43.503462 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:48:43.503463 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:48:43.503670 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:48:43.503878 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0927 12:48:43.504229 (+ 351us) server_negotiation.cc:299] Negotiation successful
0927 12:48:43.504339 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":41}
I20260927 12:48:43.504186 13943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:43.493074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45832 (local address 127.0.98.60:37657)
0927 12:48:43.493456 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0927 12:48:43.493467 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 12:48:43.493480 (+ 13us) server_negotiation.cc:407] Connection header received
0927 12:48:43.493516 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:48:43.493519 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:48:43.493708 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:48:43.493800 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:48:43.494323 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:43.495571 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:43.496577 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:43.497045 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:43.497138 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:43.500575 (+ 3437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:48:43.500580 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:48:43.500582 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:48:43.500609 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:48:43.502346 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:48:43.502804 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:48:43.502807 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:48:43.502809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:48:43.502900 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:48:43.503172 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:48:43.503175 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:48:43.503177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:48:43.503369 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:48:43.503449 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 12:48:43.503900 (+ 451us) server_negotiation.cc:299] Negotiation successful
0927 12:48:43.503992 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":317,"spinlock_wait_cycles":7296,"thread_start_us":105,"threads_started":1}
I20260927 12:48:43.507048 13787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b31d259044b44cbb67469066512838c: Bootstrap starting.
I20260927 12:48:43.507894 13938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fa0998ef5f541719a1af35b4c2c3b41: Bootstrap starting.
I20260927 12:48:43.507894 13862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 090d21945ebe484f96e97f35a88ef28e: Bootstrap starting.
I20260927 12:48:43.508014 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:48:43.509941 13787 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b31d259044b44cbb67469066512838c: Neither blocks nor log segments found. Creating new log.
I20260927 12:48:43.510565 13862 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 090d21945ebe484f96e97f35a88ef28e: Neither blocks nor log segments found. Creating new log.
I20260927 12:48:43.510565 13938 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6fa0998ef5f541719a1af35b4c2c3b41: Neither blocks nor log segments found. Creating new log.
I20260927 12:48:43.510792 13787 log.cc:824] T 00000000000000000000000000000000 P 5b31d259044b44cbb67469066512838c: Log is configured to *not* fsync() on all Append() calls
I20260927 12:48:43.511407 13938 log.cc:824] T 00000000000000000000000000000000 P 6fa0998ef5f541719a1af35b4c2c3b41: Log is configured to *not* fsync() on all Append() calls
I20260927 12:48:43.511407 13862 log.cc:824] T 00000000000000000000000000000000 P 090d21945ebe484f96e97f35a88ef28e: Log is configured to *not* fsync() on all Append() calls
I20260927 12:48:43.512821 13787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b31d259044b44cbb67469066512838c: No bootstrap required, opened a new log
I20260927 12:48:43.513427 13938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fa0998ef5f541719a1af35b4c2c3b41: No bootstrap required, opened a new log
I20260927 12:48:43.513832 13862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 090d21945ebe484f96e97f35a88ef28e: No bootstrap required, opened a new log
I20260927 12:48:43.516505 13787 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b31d259044b44cbb67469066512838c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b31d259044b44cbb67469066512838c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32849 } } peers { permanent_uuid: "090d21945ebe484f96e97f35a88ef28e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46363 } } peers { permanent_uuid: "6fa0998ef5f541719a1af35b4c2c3b41" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37657 } }
I20260927 12:48:43.516741 13787 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b31d259044b44cbb67469066512838c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:48:43.516794 13787 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b31d259044b44cbb67469066512838c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b31d259044b44cbb67469066512838c, State: Initialized, Role: FOLLOWER
I20260927 12:48:43.517218 13787 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b31d259044b44cbb67469066512838c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b31d259044b44cbb67469066512838c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32849 } } peers { permanent_uuid: "090d21945ebe484f96e97f35a88ef28e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46363 } } peers { permanent_uuid: "6fa0998ef5f541719a1af35b4c2c3b41" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37657 } }
I20260927 12:48:43.517786 13862 raft_consensus.cc:348] T 00000000000000000000000000000000 P 090d21945ebe484f96e97f35a88ef28e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b31d259044b44cbb67469066512838c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32849 } } peers { permanent_uuid: "090d21945ebe484f96e97f35a88ef28e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46363 } } peers { permanent_uuid: "6fa0998ef5f541719a1af35b4c2c3b41" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37657 } }
I20260927 12:48:43.518015 13862 raft_consensus.cc:374] T 00000000000000000000000000000000 P 090d21945ebe484f96e97f35a88ef28e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:48:43.518100 13862 raft_consensus.cc:729] T 00000000000000000000000000000000 P 090d21945ebe484f96e97f35a88ef28e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 090d21945ebe484f96e97f35a88ef28e, State: Initialized, Role: FOLLOWER
I20260927 12:48:43.518095 13938 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6fa0998ef5f541719a1af35b4c2c3b41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b31d259044b44cbb67469066512838c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32849 } } peers { permanent_uuid: "090d21945ebe484f96e97f35a88ef28e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46363 } } peers { permanent_uuid: "6fa0998ef5f541719a1af35b4c2c3b41" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37657 } }
I20260927 12:48:43.518337 13938 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6fa0998ef5f541719a1af35b4c2c3b41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:48:43.518399 13938 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6fa0998ef5f541719a1af35b4c2c3b41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fa0998ef5f541719a1af35b4c2c3b41, State: Initialized, Role: FOLLOWER
I20260927 12:48:43.518527 13787 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b31d259044b44cbb67469066512838c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:48:43.518558 13862 consensus_queue.cc:260] T 00000000000000000000000000000000 P 090d21945ebe484f96e97f35a88ef28e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b31d259044b44cbb67469066512838c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32849 } } peers { permanent_uuid: "090d21945ebe484f96e97f35a88ef28e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46363 } } peers { permanent_uuid: "6fa0998ef5f541719a1af35b4c2c3b41" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37657 } }
I20260927 12:48:43.518796 13938 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6fa0998ef5f541719a1af35b4c2c3b41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b31d259044b44cbb67469066512838c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32849 } } peers { permanent_uuid: "090d21945ebe484f96e97f35a88ef28e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46363 } } peers { permanent_uuid: "6fa0998ef5f541719a1af35b4c2c3b41" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37657 } }
I20260927 12:48:43.519109 13956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 090d21945ebe484f96e97f35a88ef28e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b31d259044b44cbb67469066512838c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32849 } } peers { permanent_uuid: "090d21945ebe484f96e97f35a88ef28e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46363 } } peers { permanent_uuid: "6fa0998ef5f541719a1af35b4c2c3b41" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37657 } } }
I20260927 12:48:43.519318 13956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 090d21945ebe484f96e97f35a88ef28e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:48:43.519894 13862 sys_catalog.cc:564] T 00000000000000000000000000000000 P 090d21945ebe484f96e97f35a88ef28e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:48:43.519894 13938 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6fa0998ef5f541719a1af35b4c2c3b41 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 12:48:43.523577 13968 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5b31d259044b44cbb67469066512838c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 12:48:43.523710 13968 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260927 12:48:43.525799 13987 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6fa0998ef5f541719a1af35b4c2c3b41: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 12:48:43.525883 13987 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 12:48:43.527215 13955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b31d259044b44cbb67469066512838c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b31d259044b44cbb67469066512838c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32849 } } peers { permanent_uuid: "090d21945ebe484f96e97f35a88ef28e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46363 } } peers { permanent_uuid: "6fa0998ef5f541719a1af35b4c2c3b41" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37657 } } }
I20260927 12:48:43.522815 13957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fa0998ef5f541719a1af35b4c2c3b41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b31d259044b44cbb67469066512838c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32849 } } peers { permanent_uuid: "090d21945ebe484f96e97f35a88ef28e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46363 } } peers { permanent_uuid: "6fa0998ef5f541719a1af35b4c2c3b41" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37657 } } }
I20260927 12:48:43.527349 13955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b31d259044b44cbb67469066512838c [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:48:43.527405 13957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fa0998ef5f541719a1af35b4c2c3b41 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 12:48:43.528662 13740 tablet.cc:2369] T 00000000000000000000000000000000 P 5b31d259044b44cbb67469066512838c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:48:43.530004 13719 master_runner.cc:263] No leader master found from master 5b31d259044b44cbb67469066512838c
I20260927 12:48:43.530107 13719 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 12:48:43.531584 13991 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 090d21945ebe484f96e97f35a88ef28e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 12:48:43.531662 13991 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 12:48:43.532027 13870 master_runner.cc:263] No leader master found from master 6fa0998ef5f541719a1af35b4c2c3b41
I20260927 12:48:43.532087 13870 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 12:48:43.532634 13794 master_runner.cc:263] No leader master found from master 090d21945ebe484f96e97f35a88ef28e
I20260927 12:48:43.532691 13794 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 12:48:43.539815 13868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:43.514662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38626 (local address 127.0.98.61:46363)
0927 12:48:43.514774 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 12:48:43.514779 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:48:43.526685 (+ 11906us) server_negotiation.cc:407] Connection header received
0927 12:48:43.528945 (+ 2260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:48:43.528949 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:48:43.529002 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:48:43.529057 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:48:43.530100 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:43.530865 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:43.534025 (+ 3160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:43.534383 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:43.534480 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:43.536026 (+ 1546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:48:43.536030 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:48:43.536033 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:48:43.536059 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:48:43.537747 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:48:43.538267 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:48:43.538271 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:48:43.538272 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:48:43.538370 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:48:43.538658 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:48:43.538661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:48:43.538663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:48:43.538870 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:48:43.538987 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 12:48:43.539400 (+ 413us) server_negotiation.cc:299] Negotiation successful
0927 12:48:43.539538 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 12:48:43.543452 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:43.512201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59418 (local address 127.0.98.62:32849)
0927 12:48:43.512341 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 12:48:43.512346 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:48:43.526300 (+ 13954us) server_negotiation.cc:407] Connection header received
0927 12:48:43.526463 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:48:43.526467 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:48:43.526516 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:48:43.526595 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:48:43.536076 (+ 9481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:43.536731 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:43.537828 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:43.538172 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:43.538244 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:43.539733 (+ 1489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:48:43.539736 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:48:43.539739 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:48:43.539767 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:48:43.541595 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:48:43.541991 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:48:43.541995 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:48:43.541997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:48:43.542105 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:48:43.542432 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:48:43.542435 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:48:43.542437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:48:43.542666 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:48:43.542777 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 12:48:43.543124 (+ 347us) server_negotiation.cc:299] Negotiation successful
0927 12:48:43.543279 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 12:48:43.544301 13944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:43.515264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45856 (local address 127.0.98.60:37657)
0927 12:48:43.515365 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0927 12:48:43.515369 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:48:43.527033 (+ 11664us) server_negotiation.cc:407] Connection header received
0927 12:48:43.527721 (+ 688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:48:43.527725 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:48:43.527773 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:48:43.527827 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:48:43.532075 (+ 4248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:43.532835 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:43.533979 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:43.534385 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:43.534488 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:43.539187 (+ 4699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:48:43.539197 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:48:43.539200 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:48:43.539231 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:48:43.541593 (+ 2362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:48:43.542495 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:48:43.542499 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:48:43.542501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:48:43.542583 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:48:43.542882 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:48:43.542885 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:48:43.542886 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:48:43.543099 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:48:43.543203 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 12:48:43.543537 (+ 334us) server_negotiation.cc:299] Negotiation successful
0927 12:48:43.543662 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 12:48:43.692641 13955 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b31d259044b44cbb67469066512838c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:48:43.692860 13955 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b31d259044b44cbb67469066512838c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b31d259044b44cbb67469066512838c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32849 } } peers { permanent_uuid: "090d21945ebe484f96e97f35a88ef28e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46363 } } peers { permanent_uuid: "6fa0998ef5f541719a1af35b4c2c3b41" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37657 } }
I20260927 12:48:43.693812 13955 leader_election.cc:288] T 00000000000000000000000000000000 P 5b31d259044b44cbb67469066512838c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 090d21945ebe484f96e97f35a88ef28e (127.0.98.61:46363), 6fa0998ef5f541719a1af35b4c2c3b41 (127.0.98.60:37657)
I20260927 12:48:43.694443 13836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5b31d259044b44cbb67469066512838c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "090d21945ebe484f96e97f35a88ef28e" is_pre_election: true
I20260927 12:48:43.694458 13912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5b31d259044b44cbb67469066512838c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fa0998ef5f541719a1af35b4c2c3b41" is_pre_election: true
I20260927 12:48:43.694777 13836 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 090d21945ebe484f96e97f35a88ef28e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b31d259044b44cbb67469066512838c in term 0.
I20260927 12:48:43.694777 13912 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6fa0998ef5f541719a1af35b4c2c3b41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b31d259044b44cbb67469066512838c in term 0.
I20260927 12:48:43.695257 13736 leader_election.cc:302] T 00000000000000000000000000000000 P 5b31d259044b44cbb67469066512838c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b31d259044b44cbb67469066512838c, 6fa0998ef5f541719a1af35b4c2c3b41; no voters:
I20260927 12:48:43.695530 13955 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b31d259044b44cbb67469066512838c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:48:43.695613 13955 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b31d259044b44cbb67469066512838c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:48:43.695653 13955 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b31d259044b44cbb67469066512838c [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:48:43.697396 13955 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b31d259044b44cbb67469066512838c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b31d259044b44cbb67469066512838c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32849 } } peers { permanent_uuid: "090d21945ebe484f96e97f35a88ef28e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46363 } } peers { permanent_uuid: "6fa0998ef5f541719a1af35b4c2c3b41" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37657 } }
I20260927 12:48:43.697733 13955 leader_election.cc:288] T 00000000000000000000000000000000 P 5b31d259044b44cbb67469066512838c [CANDIDATE]: Term 1 election: Requested vote from peers 090d21945ebe484f96e97f35a88ef28e (127.0.98.61:46363), 6fa0998ef5f541719a1af35b4c2c3b41 (127.0.98.60:37657)
I20260927 12:48:43.698259 13912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5b31d259044b44cbb67469066512838c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fa0998ef5f541719a1af35b4c2c3b41"
I20260927 12:48:43.698395 13912 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fa0998ef5f541719a1af35b4c2c3b41 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:48:43.699015 13836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5b31d259044b44cbb67469066512838c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "090d21945ebe484f96e97f35a88ef28e"
I20260927 12:48:43.699129 13836 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 090d21945ebe484f96e97f35a88ef28e [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:48:43.699698 13912 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6fa0998ef5f541719a1af35b4c2c3b41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b31d259044b44cbb67469066512838c in term 1.
I20260927 12:48:43.700016 13736 leader_election.cc:302] T 00000000000000000000000000000000 P 5b31d259044b44cbb67469066512838c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b31d259044b44cbb67469066512838c, 6fa0998ef5f541719a1af35b4c2c3b41; no voters:
I20260927 12:48:43.700403 13836 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 090d21945ebe484f96e97f35a88ef28e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b31d259044b44cbb67469066512838c in term 1.
I20260927 12:48:43.700973 13955 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b31d259044b44cbb67469066512838c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:48:43.701292 13955 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b31d259044b44cbb67469066512838c [term 1 LEADER]: Becoming Leader. State: Replica: 5b31d259044b44cbb67469066512838c, State: Running, Role: LEADER
I20260927 12:48:43.701650 13955 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b31d259044b44cbb67469066512838c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b31d259044b44cbb67469066512838c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32849 } } peers { permanent_uuid: "090d21945ebe484f96e97f35a88ef28e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46363 } } peers { permanent_uuid: "6fa0998ef5f541719a1af35b4c2c3b41" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37657 } }
I20260927 12:48:43.703475 13994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b31d259044b44cbb67469066512838c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b31d259044b44cbb67469066512838c. Latest consensus state: current_term: 1 leader_uuid: "5b31d259044b44cbb67469066512838c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b31d259044b44cbb67469066512838c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32849 } } peers { permanent_uuid: "090d21945ebe484f96e97f35a88ef28e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46363 } } peers { permanent_uuid: "6fa0998ef5f541719a1af35b4c2c3b41" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37657 } } }
I20260927 12:48:43.703572 13994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b31d259044b44cbb67469066512838c [sys.catalog]: This master's current role is: LEADER
I20260927 12:48:43.703943 13996 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:48:43.704908 13996 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260927 12:48:43.706216 13891 tablet.cc:2369] T 00000000000000000000000000000000 P 6fa0998ef5f541719a1af35b4c2c3b41: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:48:43.709523 13836 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 090d21945ebe484f96e97f35a88ef28e [term 1 FOLLOWER]: Refusing update from remote peer 5b31d259044b44cbb67469066512838c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:48:43.709523 13912 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6fa0998ef5f541719a1af35b4c2c3b41 [term 1 FOLLOWER]: Refusing update from remote peer 5b31d259044b44cbb67469066512838c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:48:43.710239 13994 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5b31d259044b44cbb67469066512838c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fa0998ef5f541719a1af35b4c2c3b41" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:48:43.710479 13955 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5b31d259044b44cbb67469066512838c [LEADER]: Connected to new peer: Peer: permanent_uuid: "090d21945ebe484f96e97f35a88ef28e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:48:43.714393 13956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 090d21945ebe484f96e97f35a88ef28e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b31d259044b44cbb67469066512838c. Latest consensus state: current_term: 1 leader_uuid: "5b31d259044b44cbb67469066512838c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b31d259044b44cbb67469066512838c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32849 } } peers { permanent_uuid: "090d21945ebe484f96e97f35a88ef28e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46363 } } peers { permanent_uuid: "6fa0998ef5f541719a1af35b4c2c3b41" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37657 } } }
I20260927 12:48:43.714541 13956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 090d21945ebe484f96e97f35a88ef28e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:48:43.714973 13957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fa0998ef5f541719a1af35b4c2c3b41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b31d259044b44cbb67469066512838c. Latest consensus state: current_term: 1 leader_uuid: "5b31d259044b44cbb67469066512838c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b31d259044b44cbb67469066512838c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32849 } } peers { permanent_uuid: "090d21945ebe484f96e97f35a88ef28e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46363 } } peers { permanent_uuid: "6fa0998ef5f541719a1af35b4c2c3b41" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37657 } } }
I20260927 12:48:43.715090 13957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fa0998ef5f541719a1af35b4c2c3b41 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:48:43.721032 13997 mvcc.cc:205] Tried to move back new op lower bound from 7333942573907476480 to 7333942573884776448. Current Snapshot: MvccSnapshot[applied={T|T < 7333942573907476480}]
I20260927 12:48:43.720973 13955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b31d259044b44cbb67469066512838c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5b31d259044b44cbb67469066512838c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b31d259044b44cbb67469066512838c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32849 } } peers { permanent_uuid: "090d21945ebe484f96e97f35a88ef28e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46363 } } peers { permanent_uuid: "6fa0998ef5f541719a1af35b4c2c3b41" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37657 } } }
I20260927 12:48:43.721115 13955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b31d259044b44cbb67469066512838c [sys.catalog]: This master's current role is: LEADER
I20260927 12:48:43.722124 13999 mvcc.cc:205] Tried to move back new op lower bound from 7333942573907476480 to 7333942573884776448. Current Snapshot: MvccSnapshot[applied={T|T < 7333942573907476480}]
I20260927 12:48:43.723186 13994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b31d259044b44cbb67469066512838c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5b31d259044b44cbb67469066512838c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b31d259044b44cbb67469066512838c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32849 } } peers { permanent_uuid: "090d21945ebe484f96e97f35a88ef28e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46363 } } peers { permanent_uuid: "6fa0998ef5f541719a1af35b4c2c3b41" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37657 } } }
I20260927 12:48:43.723297 13994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b31d259044b44cbb67469066512838c [sys.catalog]: This master's current role is: LEADER
I20260927 12:48:43.723438 13996 catalog_manager.cc:1347] Generated new cluster ID: 60f1d26a69284d219f5aecc737f9405c
I20260927 12:48:43.723479 13996 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:48:43.724498 13998 mvcc.cc:205] Tried to move back new op lower bound from 7333942573907476480 to 7333942573884776448. Current Snapshot: MvccSnapshot[applied={T|T < 7333942573907476480}]
I20260927 12:48:43.725239 13956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 090d21945ebe484f96e97f35a88ef28e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5b31d259044b44cbb67469066512838c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b31d259044b44cbb67469066512838c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32849 } } peers { permanent_uuid: "090d21945ebe484f96e97f35a88ef28e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46363 } } peers { permanent_uuid: "6fa0998ef5f541719a1af35b4c2c3b41" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37657 } } }
I20260927 12:48:43.725369 13956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 090d21945ebe484f96e97f35a88ef28e [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:48:43.725971 13957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fa0998ef5f541719a1af35b4c2c3b41 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5b31d259044b44cbb67469066512838c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b31d259044b44cbb67469066512838c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32849 } } peers { permanent_uuid: "090d21945ebe484f96e97f35a88ef28e" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46363 } } peers { permanent_uuid: "6fa0998ef5f541719a1af35b4c2c3b41" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37657 } } }
I20260927 12:48:43.726099 13957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fa0998ef5f541719a1af35b4c2c3b41 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:48:43.742502 13996 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:48:43.743773 13996 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:48:43.755564 13996 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b31d259044b44cbb67469066512838c: Generated new TSK 0
I20260927 12:48:43.756481 13996 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:48:44.527217 13987 catalog_manager.cc:1259] Loaded cluster ID: 60f1d26a69284d219f5aecc737f9405c
I20260927 12:48:44.527311 13987 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6fa0998ef5f541719a1af35b4c2c3b41: loading cluster ID for follower catalog manager: success
I20260927 12:48:44.529002 13987 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6fa0998ef5f541719a1af35b4c2c3b41: acquiring CA information for follower catalog manager: success
I20260927 12:48:44.530261 13987 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6fa0998ef5f541719a1af35b4c2c3b41: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 12:48:44.532836 13991 catalog_manager.cc:1259] Loaded cluster ID: 60f1d26a69284d219f5aecc737f9405c
I20260927 12:48:44.532894 13991 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 090d21945ebe484f96e97f35a88ef28e: loading cluster ID for follower catalog manager: success
I20260927 12:48:44.534534 13991 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 090d21945ebe484f96e97f35a88ef28e: acquiring CA information for follower catalog manager: success
I20260927 12:48:44.535605 13991 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 090d21945ebe484f96e97f35a88ef28e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 12:48:44.810513 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 13719
I20260927 12:48:44.815968 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 13794
I20260927 12:48:44.820711 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 13870
2026-09-27T12:48:44Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3049 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790513161160093-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:48:44 dist-test-slave-lkn4 krb5kdc[14011](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:48:44 dist-test-slave-lkn4 krb5kdc[14011](info): set up 2 sockets
Sep 27 12:48:44 dist-test-slave-lkn4 krb5kdc[14011](info): commencing operation
krb5kdc: starting...
W20260927 12:48:45.778131 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790513161160093-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790513161160093-392-0/krb5kdc/test-user.keytab.
Sep 27 12:48:45 dist-test-slave-lkn4 krb5kdc[14011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:48:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:48:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:48:45.897145 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790513161160093-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790513161160093-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790513161160093-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41315
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41315
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:48:46.005386 14024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:48:46.005678 14024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:48:46.005736 14024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:48:46.009753 14024 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:48:46.009828 14024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:48:46.009855 14024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:48:46.009886 14024 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:48:46.009904 14024 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:48:46.015328 14024 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790513161160093-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790513161160093-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41315
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790513161160093-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:48:46.016439 14024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:48:46.017468 14024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:48:46.025758 14032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:48:46.025903 14030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:48:46.025861 14024 server_base.cc:1034] running on GCE node
W20260927 12:48:46.025789 14029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:48:46.026672 14024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:48:46.027746 14024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:48:46.028918 14024 hybrid_clock.cc:648] HybridClock initialized: now 1790513326028899 us; error 32 us; skew 500 ppm
Sep 27 12:48:46 dist-test-slave-lkn4 krb5kdc[14011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513326, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:48:46.031906 14024 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:48:46.032887 14024 webserver.cc:460] Webserver started at http://127.0.98.62:44775/ using document root <none> and password file <none>
I20260927 12:48:46.033432 14024 fs_manager.cc:359] Metadata directory not provided
I20260927 12:48:46.033494 14024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:48:46.033701 14024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:48:46.035881 14024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790513161160093-392-0/minicluster-data/master-0/data/instance:
uuid: "fb8fdad73f4f4199a553d2ce0ca6f92c"
format_stamp: "Formatted at 2026-09-27 12:48:46 on dist-test-slave-lkn4"
server_key: "70842a5fb5aded3cb6f9980f37e2a42d"
server_key_iv: "9fd9878dcdb84182108c1acbf626c6bc"
server_key_version: "encryptionkey@0"
I20260927 12:48:46.036424 14024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790513161160093-392-0/minicluster-data/master-0/wal/instance:
uuid: "fb8fdad73f4f4199a553d2ce0ca6f92c"
format_stamp: "Formatted at 2026-09-27 12:48:46 on dist-test-slave-lkn4"
server_key: "70842a5fb5aded3cb6f9980f37e2a42d"
server_key_iv: "9fd9878dcdb84182108c1acbf626c6bc"
server_key_version: "encryptionkey@0"
I20260927 12:48:46.040244 14024 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 12:48:46.042891 14039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:48:46.044236 14024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:48:46.044360 14024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790513161160093-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790513161160093-392-0/minicluster-data/master-0/wal
uuid: "fb8fdad73f4f4199a553d2ce0ca6f92c"
format_stamp: "Formatted at 2026-09-27 12:48:46 on dist-test-slave-lkn4"
server_key: "70842a5fb5aded3cb6f9980f37e2a42d"
server_key_iv: "9fd9878dcdb84182108c1acbf626c6bc"
server_key_version: "encryptionkey@0"
I20260927 12:48:46.044482 14024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790513161160093-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790513161160093-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790513161160093-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:48:46.060233 14024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:48:46.063601 14024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:48:46.063769 14024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:48:46.070358 14024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41315
I20260927 12:48:46.070393 14091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41315 every 8 connection(s)
I20260927 12:48:46.071316 14024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:48:46.073132 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 14024
I20260927 12:48:46.073251 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790513161160093-392-0/minicluster-data/master-0/wal/instance
I20260927 12:48:46.073573 392 external_mini_cluster.cc:1428] Setting key 5aae00759f87c7169cd3b2251dc88e07
I20260927 12:48:46.074375 14092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:48:46.079906 14092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb8fdad73f4f4199a553d2ce0ca6f92c: Bootstrap starting.
Sep 27 12:48:46 dist-test-slave-lkn4 krb5kdc[14011](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:48:46.082258 14092 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb8fdad73f4f4199a553d2ce0ca6f92c: Neither blocks nor log segments found. Creating new log.
I20260927 12:48:46.083097 14092 log.cc:824] T 00000000000000000000000000000000 P fb8fdad73f4f4199a553d2ce0ca6f92c: Log is configured to *not* fsync() on all Append() calls
I20260927 12:48:46.084971 14092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb8fdad73f4f4199a553d2ce0ca6f92c: No bootstrap required, opened a new log
I20260927 12:48:46.086978 14095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:46.074914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37222 (local address 127.0.98.62:41315)
0927 12:48:46.075347 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0927 12:48:46.075363 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 12:48:46.075397 (+ 34us) server_negotiation.cc:407] Connection header received
0927 12:48:46.076014 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:48:46.076030 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:48:46.076360 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:48:46.076664 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:48:46.077477 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:46.078515 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:46.079338 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:46.079798 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:46.079948 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:46.082529 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:48:46.082535 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:48:46.082552 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:48:46.082584 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:48:46.084215 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:48:46.084617 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:48:46.084622 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:48:46.084626 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:48:46.084735 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:48:46.085026 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:48:46.085029 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:48:46.085031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:48:46.085457 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:48:46.085632 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 12:48:46.085908 (+ 276us) server_negotiation.cc:299] Negotiation successful
0927 12:48:46.086223 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":138,"threads_started":1}
I20260927 12:48:46.088785 14092 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb8fdad73f4f4199a553d2ce0ca6f92c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb8fdad73f4f4199a553d2ce0ca6f92c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41315 } }
I20260927 12:48:46.089061 14092 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb8fdad73f4f4199a553d2ce0ca6f92c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:48:46.089129 14092 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb8fdad73f4f4199a553d2ce0ca6f92c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb8fdad73f4f4199a553d2ce0ca6f92c, State: Initialized, Role: FOLLOWER
I20260927 12:48:46.089507 14092 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb8fdad73f4f4199a553d2ce0ca6f92c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb8fdad73f4f4199a553d2ce0ca6f92c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41315 } }
I20260927 12:48:46.089632 14092 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb8fdad73f4f4199a553d2ce0ca6f92c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:48:46.089692 14092 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb8fdad73f4f4199a553d2ce0ca6f92c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:48:46.089789 14092 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb8fdad73f4f4199a553d2ce0ca6f92c [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:48:46.089780 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:48:46.090776 14092 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb8fdad73f4f4199a553d2ce0ca6f92c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb8fdad73f4f4199a553d2ce0ca6f92c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41315 } }
I20260927 12:48:46.091120 14092 leader_election.cc:302] T 00000000000000000000000000000000 P fb8fdad73f4f4199a553d2ce0ca6f92c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb8fdad73f4f4199a553d2ce0ca6f92c; no voters:
I20260927 12:48:46.091465 14092 leader_election.cc:288] T 00000000000000000000000000000000 P fb8fdad73f4f4199a553d2ce0ca6f92c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:48:46.091560 14097 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb8fdad73f4f4199a553d2ce0ca6f92c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:48:46.091778 14097 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb8fdad73f4f4199a553d2ce0ca6f92c [term 1 LEADER]: Becoming Leader. State: Replica: fb8fdad73f4f4199a553d2ce0ca6f92c, State: Running, Role: LEADER
I20260927 12:48:46.092140 14097 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb8fdad73f4f4199a553d2ce0ca6f92c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb8fdad73f4f4199a553d2ce0ca6f92c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41315 } }
I20260927 12:48:46.092798 14092 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb8fdad73f4f4199a553d2ce0ca6f92c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:48:46.093737 14098 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb8fdad73f4f4199a553d2ce0ca6f92c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb8fdad73f4f4199a553d2ce0ca6f92c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb8fdad73f4f4199a553d2ce0ca6f92c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41315 } } }
I20260927 12:48:46.093741 14099 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb8fdad73f4f4199a553d2ce0ca6f92c [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb8fdad73f4f4199a553d2ce0ca6f92c. Latest consensus state: current_term: 1 leader_uuid: "fb8fdad73f4f4199a553d2ce0ca6f92c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb8fdad73f4f4199a553d2ce0ca6f92c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41315 } } }
I20260927 12:48:46.093914 14098 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb8fdad73f4f4199a553d2ce0ca6f92c [sys.catalog]: This master's current role is: LEADER
I20260927 12:48:46.093941 14099 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb8fdad73f4f4199a553d2ce0ca6f92c [sys.catalog]: This master's current role is: LEADER
W20260927 12:48:46.097447 14113 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fb8fdad73f4f4199a553d2ce0ca6f92c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 12:48:46.097517 14113 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 12:48:46.097599 14106 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:48:46.098552 14106 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:48:46.104028 14106 catalog_manager.cc:1347] Generated new cluster ID: 7a8615aefd484eef9004f0f780ada57b
I20260927 12:48:46.104100 14106 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:48:46.150816 14106 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:48:46.152168 14106 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:48:46.157946 14106 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb8fdad73f4f4199a553d2ce0ca6f92c: Generated new TSK 0
I20260927 12:48:46.158809 14106 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:48:46.196874 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 14024
2026-09-27T12:48:46Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1375 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:48:46 dist-test-slave-lkn4 krb5kdc[14119](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:48:46 dist-test-slave-lkn4 krb5kdc[14119](info): set up 2 sockets
Sep 27 12:48:46 dist-test-slave-lkn4 krb5kdc[14119](info): commencing operation
krb5kdc: starting...
W20260927 12:48:47.170107 392 mini_kdc.cc:121] Time spent starting KDC: real 0.954s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/krb5kdc/test-user.keytab.
Sep 27 12:48:47 dist-test-slave-lkn4 krb5kdc[14119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:48:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:48:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:48:47.289824 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37309
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37309
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:48:47.397243 14132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:48:47.397538 14132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:48:47.397579 14132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:48:47.401537 14132 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:48:47.401595 14132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:48:47.401613 14132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:48:47.401629 14132 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:48:47.401643 14132 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:48:47.406927 14132 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37309
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:48:47.407923 14132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:48:47.408877 14132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:48:47.417366 14137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:48:47.417387 14138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:48:47.417404 14140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:48:47.417383 14132 server_base.cc:1034] running on GCE node
I20260927 12:48:47.418323 14132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:48:47.419539 14132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:48:47.420738 14132 hybrid_clock.cc:648] HybridClock initialized: now 1790513327420715 us; error 44 us; skew 500 ppm
Sep 27 12:48:47 dist-test-slave-lkn4 krb5kdc[14119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:48:47.429435 14132 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:48:47.430497 14132 webserver.cc:460] Webserver started at http://127.0.98.62:44923/ using document root <none> and password file <none>
I20260927 12:48:47.431114 14132 fs_manager.cc:359] Metadata directory not provided
I20260927 12:48:47.431187 14132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:48:47.431394 14132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:48:47.433493 14132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/master-0/data/instance:
uuid: "76a78d2ebcd0446ebdafa41ff473c402"
format_stamp: "Formatted at 2026-09-27 12:48:47 on dist-test-slave-lkn4"
server_key: "f7aafd8aa5d0a262908a480aa93c0a1f"
server_key_iv: "b340ad9cad868032c70ae03cb94d9054"
server_key_version: "encryptionkey@0"
I20260927 12:48:47.434068 14132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/master-0/wal/instance:
uuid: "76a78d2ebcd0446ebdafa41ff473c402"
format_stamp: "Formatted at 2026-09-27 12:48:47 on dist-test-slave-lkn4"
server_key: "f7aafd8aa5d0a262908a480aa93c0a1f"
server_key_iv: "b340ad9cad868032c70ae03cb94d9054"
server_key_version: "encryptionkey@0"
I20260927 12:48:47.437958 14132 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 12:48:47.440538 14147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:48:47.441969 14132 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 12:48:47.442128 14132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/master-0/wal
uuid: "76a78d2ebcd0446ebdafa41ff473c402"
format_stamp: "Formatted at 2026-09-27 12:48:47 on dist-test-slave-lkn4"
server_key: "f7aafd8aa5d0a262908a480aa93c0a1f"
server_key_iv: "b340ad9cad868032c70ae03cb94d9054"
server_key_version: "encryptionkey@0"
I20260927 12:48:47.442237 14132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:48:47.461817 14132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:48:47.465047 14132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:48:47.465227 14132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:48:47.472750 14132 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37309
I20260927 12:48:47.473057 14199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37309 every 8 connection(s)
I20260927 12:48:47.473757 14132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:48:47.475929 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 14132
I20260927 12:48:47.476039 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/master-0/wal/instance
I20260927 12:48:47.476289 392 external_mini_cluster.cc:1428] Setting key dd80d7a08ffa8848baa0622083162035
I20260927 12:48:47.476645 14200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:48:47.483150 14200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76a78d2ebcd0446ebdafa41ff473c402: Bootstrap starting.
Sep 27 12:48:47 dist-test-slave-lkn4 krb5kdc[14119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:48:47.485805 14200 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76a78d2ebcd0446ebdafa41ff473c402: Neither blocks nor log segments found. Creating new log.
I20260927 12:48:47.486544 14200 log.cc:824] T 00000000000000000000000000000000 P 76a78d2ebcd0446ebdafa41ff473c402: Log is configured to *not* fsync() on all Append() calls
I20260927 12:48:47.488659 14200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76a78d2ebcd0446ebdafa41ff473c402: No bootstrap required, opened a new log
I20260927 12:48:47.490563 14203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:47.477591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45244 (local address 127.0.98.62:37309)
0927 12:48:47.477975 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0927 12:48:47.477991 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 12:48:47.478027 (+ 36us) server_negotiation.cc:407] Connection header received
0927 12:48:47.478702 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:48:47.478719 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:48:47.479051 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:48:47.479322 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:48:47.480238 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:47.481338 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:47.482494 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:47.482963 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:47.483075 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:47.485670 (+ 2595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:48:47.485674 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:48:47.485687 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:48:47.485713 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:48:47.487456 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:48:47.487931 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:48:47.487937 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:48:47.487942 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:48:47.488059 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:48:47.488340 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:48:47.488342 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:48:47.488344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:48:47.488813 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:48:47.488988 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 12:48:47.489250 (+ 262us) server_negotiation.cc:299] Negotiation successful
0927 12:48:47.489529 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":107,"threads_started":1}
I20260927 12:48:47.491854 14200 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76a78d2ebcd0446ebdafa41ff473c402 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76a78d2ebcd0446ebdafa41ff473c402" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37309 } }
I20260927 12:48:47.492074 14200 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76a78d2ebcd0446ebdafa41ff473c402 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:48:47.492126 14200 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76a78d2ebcd0446ebdafa41ff473c402 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76a78d2ebcd0446ebdafa41ff473c402, State: Initialized, Role: FOLLOWER
I20260927 12:48:47.492527 14200 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76a78d2ebcd0446ebdafa41ff473c402 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76a78d2ebcd0446ebdafa41ff473c402" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37309 } }
I20260927 12:48:47.492651 14200 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76a78d2ebcd0446ebdafa41ff473c402 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:48:47.492739 14200 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76a78d2ebcd0446ebdafa41ff473c402 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:48:47.492821 14200 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76a78d2ebcd0446ebdafa41ff473c402 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:48:47.493659 14200 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76a78d2ebcd0446ebdafa41ff473c402 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76a78d2ebcd0446ebdafa41ff473c402" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37309 } }
I20260927 12:48:47.493981 14200 leader_election.cc:302] T 00000000000000000000000000000000 P 76a78d2ebcd0446ebdafa41ff473c402 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76a78d2ebcd0446ebdafa41ff473c402; no voters:
I20260927 12:48:47.494270 14200 leader_election.cc:288] T 00000000000000000000000000000000 P 76a78d2ebcd0446ebdafa41ff473c402 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:48:47.494374 14205 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76a78d2ebcd0446ebdafa41ff473c402 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:48:47.494581 14205 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76a78d2ebcd0446ebdafa41ff473c402 [term 1 LEADER]: Becoming Leader. State: Replica: 76a78d2ebcd0446ebdafa41ff473c402, State: Running, Role: LEADER
I20260927 12:48:47.494936 14205 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76a78d2ebcd0446ebdafa41ff473c402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76a78d2ebcd0446ebdafa41ff473c402" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37309 } }
I20260927 12:48:47.495385 14200 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76a78d2ebcd0446ebdafa41ff473c402 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:48:47.496425 14207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76a78d2ebcd0446ebdafa41ff473c402 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76a78d2ebcd0446ebdafa41ff473c402. Latest consensus state: current_term: 1 leader_uuid: "76a78d2ebcd0446ebdafa41ff473c402" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76a78d2ebcd0446ebdafa41ff473c402" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37309 } } }
I20260927 12:48:47.496804 14207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76a78d2ebcd0446ebdafa41ff473c402 [sys.catalog]: This master's current role is: LEADER
I20260927 12:48:47.496685 14206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76a78d2ebcd0446ebdafa41ff473c402 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76a78d2ebcd0446ebdafa41ff473c402" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76a78d2ebcd0446ebdafa41ff473c402" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37309 } } }
I20260927 12:48:47.497063 14206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76a78d2ebcd0446ebdafa41ff473c402 [sys.catalog]: This master's current role is: LEADER
W20260927 12:48:47.499848 14220 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 76a78d2ebcd0446ebdafa41ff473c402: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 12:48:47.499909 14220 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 12:48:47.500007 14218 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:48:47.500733 14218 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:48:47.505729 14218 catalog_manager.cc:1347] Generated new cluster ID: 6c455548fd1a40088a72a562cfb393ba
I20260927 12:48:47.505808 14218 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:48:47.519677 14218 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:48:47.520653 14218 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:48:47.530890 14218 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76a78d2ebcd0446ebdafa41ff473c402: Generated new TSK 0
I20260927 12:48:47.531606 14218 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 12:48:47.569406 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37309
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:48:47.678697 14226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:48:47.679071 14226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:48:47.679139 14226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:48:47.683493 14226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:48:47.683566 14226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:48:47.683625 14226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:48:47.689111 14226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14226
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:48:47.690218 14226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:48:47.691231 14226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:48:47.700778 14232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:48:47.700762 14231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:48:47.700762 14234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:48:47.701114 14226 server_base.cc:1034] running on GCE node
I20260927 12:48:47.701484 14226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:48:47.702162 14226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:48:47.703323 14226 hybrid_clock.cc:648] HybridClock initialized: now 1790513327703304 us; error 31 us; skew 500 ppm
Sep 27 12:48:47 dist-test-slave-lkn4 krb5kdc[14119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:48:47.705996 14226 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 12:48:47.706876 14226 webserver.cc:460] Webserver started at http://127.0.98.1:43289/ using document root <none> and password file <none>
I20260927 12:48:47.707412 14226 fs_manager.cc:359] Metadata directory not provided
I20260927 12:48:47.707469 14226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:48:47.707652 14226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:48:47.709813 14226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-0/data/instance:
uuid: "104cf2fdae024d4ebe3c93c3ea1afe37"
format_stamp: "Formatted at 2026-09-27 12:48:47 on dist-test-slave-lkn4"
server_key: "7999efb846f0c33cee2dd92b36225e5d"
server_key_iv: "25a33bbfb862beab983e0ff1d9a486a5"
server_key_version: "encryptionkey@0"
I20260927 12:48:47.710392 14226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-0/wal/instance:
uuid: "104cf2fdae024d4ebe3c93c3ea1afe37"
format_stamp: "Formatted at 2026-09-27 12:48:47 on dist-test-slave-lkn4"
server_key: "7999efb846f0c33cee2dd92b36225e5d"
server_key_iv: "25a33bbfb862beab983e0ff1d9a486a5"
server_key_version: "encryptionkey@0"
I20260927 12:48:47.714254 14226 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 12:48:47.716646 14241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:48:47.717900 14226 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:48:47.717995 14226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-0/wal
uuid: "104cf2fdae024d4ebe3c93c3ea1afe37"
format_stamp: "Formatted at 2026-09-27 12:48:47 on dist-test-slave-lkn4"
server_key: "7999efb846f0c33cee2dd92b36225e5d"
server_key_iv: "25a33bbfb862beab983e0ff1d9a486a5"
server_key_version: "encryptionkey@0"
I20260927 12:48:47.718143 14226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:48:47.733804 14226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:48:47.737361 14226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:48:47.737550 14226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:48:47.738178 14226 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:48:47.739004 14226 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:48:47.739050 14226 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:48:47.739090 14226 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:48:47.739117 14226 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:48:47.747737 14226 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37565
I20260927 12:48:47.747773 14354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37565 every 8 connection(s)
I20260927 12:48:47.748611 14226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:48:47.755833 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 14226
I20260927 12:48:47.755962 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:48:47.756227 392 external_mini_cluster.cc:1428] Setting key 53b3c5926cdae916c407f3011c087477
Sep 27 12:48:47 dist-test-slave-lkn4 krb5kdc[14119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:48:47.762125 14203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:47.750469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53087 (local address 127.0.98.62:37309)
0927 12:48:47.750608 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 12:48:47.750612 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:48:47.751637 (+ 1025us) server_negotiation.cc:407] Connection header received
0927 12:48:47.752569 (+ 932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:48:47.752573 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:48:47.752632 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:48:47.752705 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:48:47.753973 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:47.754578 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:47.755713 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:47.755966 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:47.756064 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:47.758729 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:48:47.758733 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:48:47.758735 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:48:47.758765 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:48:47.760150 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:48:47.760639 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:48:47.760642 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:48:47.760644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:48:47.760721 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:48:47.761037 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:48:47.761040 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:48:47.761042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:48:47.761217 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:48:47.761300 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 12:48:47.761805 (+ 505us) server_negotiation.cc:299] Negotiation successful
0927 12:48:47.761932 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 12:48:47.763028 14357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:47.750828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37309 (local address 127.0.98.1:53087)
0927 12:48:47.751466 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:47.751526 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 12:48:47.752350 (+ 824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:48:47.752958 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:48:47.752965 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:48:47.753339 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:48:47.753828 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:47.753843 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:47.754707 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:47.754711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:47.755599 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:47.755609 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:47.756128 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:47.756132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:47.756290 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:47.756866 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:48:47.756898 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:48:47.758552 (+ 1654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:48:47.760255 (+ 1703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:48:47.760260 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:48:47.760275 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:48:47.760545 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:48:47.760800 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:48:47.760803 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:48:47.760805 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:48:47.760955 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:48:47.761317 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:48:47.761323 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:48:47.761575 (+ 252us) client_negotiation.cc:769] Sending connection context
0927 12:48:47.761742 (+ 167us) client_negotiation.cc:240] Negotiation successful
0927 12:48:47.762133 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":108,"threads_started":1}
I20260927 12:48:47.764492 14355 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37309
I20260927 12:48:47.764724 14355 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 12:48:47.765230 14355 heartbeater.cc:511] Master 127.0.98.62:37309 requested a full tablet report, sending...
I20260927 12:48:47.766856 14164 ts_manager.cc:194] Registered new tserver with Master: 104cf2fdae024d4ebe3c93c3ea1afe37 (127.0.98.1:37565)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 12:48:47.768133 14164 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53087
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 12:48:47.785130 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37309
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:48:47.893179 14360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:48:47.893432 14360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:48:47.893472 14360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:48:47.897768 14360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:48:47.897825 14360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:48:47.897892 14360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:48:47.903033 14360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:48:47.904045 14360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:48:47.905012 14360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:48:47.915069 14368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:48:47.915199 14366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:48:47.915259 14360 server_base.cc:1034] running on GCE node
W20260927 12:48:47.915465 14365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:48:47.915860 14360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:48:47.916510 14360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:48:47.917690 14360 hybrid_clock.cc:648] HybridClock initialized: now 1790513327917669 us; error 31 us; skew 500 ppm
Sep 27 12:48:47 dist-test-slave-lkn4 krb5kdc[14119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:48:47.920528 14360 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 12:48:47.921411 14360 webserver.cc:460] Webserver started at http://127.0.98.2:46479/ using document root <none> and password file <none>
I20260927 12:48:47.921911 14360 fs_manager.cc:359] Metadata directory not provided
I20260927 12:48:47.921964 14360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:48:47.922169 14360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:48:47.924265 14360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-1/data/instance:
uuid: "8cbcb48cd2ab4a93956f5392df6c5983"
format_stamp: "Formatted at 2026-09-27 12:48:47 on dist-test-slave-lkn4"
server_key: "6050048a83049ae6e551f24e3f21eab9"
server_key_iv: "d51ae3f054663ff1319b0f5dec057945"
server_key_version: "encryptionkey@0"
I20260927 12:48:47.924804 14360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8cbcb48cd2ab4a93956f5392df6c5983"
format_stamp: "Formatted at 2026-09-27 12:48:47 on dist-test-slave-lkn4"
server_key: "6050048a83049ae6e551f24e3f21eab9"
server_key_iv: "d51ae3f054663ff1319b0f5dec057945"
server_key_version: "encryptionkey@0"
I20260927 12:48:47.928606 14360 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260927 12:48:47.931195 14375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:48:47.932464 14360 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:48:47.932559 14360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-1/wal
uuid: "8cbcb48cd2ab4a93956f5392df6c5983"
format_stamp: "Formatted at 2026-09-27 12:48:47 on dist-test-slave-lkn4"
server_key: "6050048a83049ae6e551f24e3f21eab9"
server_key_iv: "d51ae3f054663ff1319b0f5dec057945"
server_key_version: "encryptionkey@0"
I20260927 12:48:47.932668 14360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:48:47.947638 14360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:48:47.950987 14360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:48:47.951177 14360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:48:47.951766 14360 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:48:47.952711 14360 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:48:47.952757 14360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:48:47.952798 14360 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:48:47.952812 14360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:48:47.961447 14360 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39543
I20260927 12:48:47.961477 14488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39543 every 8 connection(s)
I20260927 12:48:47.962363 14360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:48:47.971254 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 14360
I20260927 12:48:47.971375 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:48:47.971783 392 external_mini_cluster.cc:1428] Setting key 4a7a2ea0a92eb0cccf7bd864150bc093
Sep 27 12:48:47 dist-test-slave-lkn4 krb5kdc[14119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:48:47.975934 14203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:47.964288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47381 (local address 127.0.98.62:37309)
0927 12:48:47.964421 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 12:48:47.964425 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:48:47.965487 (+ 1062us) server_negotiation.cc:407] Connection header received
0927 12:48:47.966418 (+ 931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:48:47.966421 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:48:47.966469 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:48:47.966543 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:48:47.967765 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:47.968303 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:47.969388 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:47.969721 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:47.969830 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:47.972570 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:48:47.972573 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:48:47.972575 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:48:47.972602 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:48:47.973942 (+ 1340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:48:47.974462 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:48:47.974466 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:48:47.974467 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:48:47.974541 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:48:47.974876 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:48:47.974878 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:48:47.974880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:48:47.975051 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:48:47.975135 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 12:48:47.975648 (+ 513us) server_negotiation.cc:299] Negotiation successful
0927 12:48:47.975774 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 12:48:47.976778 14491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:47.964651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37309 (local address 127.0.98.2:47381)
0927 12:48:47.965319 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:47.965361 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 12:48:47.966196 (+ 835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:48:47.966747 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:48:47.966763 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:48:47.967122 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:48:47.967620 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:47.967633 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:47.968394 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:47.968396 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:47.969265 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:47.969274 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:47.969894 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:47.969897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:47.970086 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:47.970697 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:48:47.970717 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:48:47.972396 (+ 1679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:48:47.974080 (+ 1684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:48:47.974086 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:48:47.974108 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:48:47.974368 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:48:47.974620 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:48:47.974623 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:48:47.974626 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:48:47.974781 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:48:47.975148 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:48:47.975155 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:48:47.975411 (+ 256us) client_negotiation.cc:769] Sending connection context
0927 12:48:47.975575 (+ 164us) client_negotiation.cc:240] Negotiation successful
0927 12:48:47.975920 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":110,"threads_started":1}
I20260927 12:48:47.978020 14489 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37309
I20260927 12:48:47.978298 14489 heartbeater.cc:464] Registering TS with master...
I20260927 12:48:47.978790 14489 heartbeater.cc:511] Master 127.0.98.62:37309 requested a full tablet report, sending...
I20260927 12:48:47.979914 14164 ts_manager.cc:194] Registered new tserver with Master: 8cbcb48cd2ab4a93956f5392df6c5983 (127.0.98.2:39543)
I20260927 12:48:47.980477 14164 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47381
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 12:48:47.999282 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37309
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:48:48.108664 14494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:48:48.108922 14494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:48:48.108963 14494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:48:48.113093 14494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:48:48.113155 14494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:48:48.113220 14494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:48:48.118510 14494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:48:48.119508 14494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:48:48.120474 14494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:48:48.130364 14500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:48:48.130366 14502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:48:48.130366 14499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:48:48.130848 14494 server_base.cc:1034] running on GCE node
I20260927 12:48:48.131281 14494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:48:48.131933 14494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:48:48.133106 14494 hybrid_clock.cc:648] HybridClock initialized: now 1790513328133083 us; error 37 us; skew 500 ppm
Sep 27 12:48:48 dist-test-slave-lkn4 krb5kdc[14119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513328, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:48:48.136032 14494 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 12:48:48.136960 14494 webserver.cc:460] Webserver started at http://127.0.98.3:35697/ using document root <none> and password file <none>
I20260927 12:48:48.137517 14494 fs_manager.cc:359] Metadata directory not provided
I20260927 12:48:48.137580 14494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:48:48.137789 14494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:48:48.139837 14494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-2/data/instance:
uuid: "87c9851a903c4436b895d7da878979c7"
format_stamp: "Formatted at 2026-09-27 12:48:48 on dist-test-slave-lkn4"
server_key: "c8ecfc2f6ec145221221b98795cc6cee"
server_key_iv: "424022327d0b2b00281f1069faf133aa"
server_key_version: "encryptionkey@0"
I20260927 12:48:48.140360 14494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-2/wal/instance:
uuid: "87c9851a903c4436b895d7da878979c7"
format_stamp: "Formatted at 2026-09-27 12:48:48 on dist-test-slave-lkn4"
server_key: "c8ecfc2f6ec145221221b98795cc6cee"
server_key_iv: "424022327d0b2b00281f1069faf133aa"
server_key_version: "encryptionkey@0"
I20260927 12:48:48.143926 14494 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 12:48:48.146296 14509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:48:48.147526 14494 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:48:48.147624 14494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-2/wal
uuid: "87c9851a903c4436b895d7da878979c7"
format_stamp: "Formatted at 2026-09-27 12:48:48 on dist-test-slave-lkn4"
server_key: "c8ecfc2f6ec145221221b98795cc6cee"
server_key_iv: "424022327d0b2b00281f1069faf133aa"
server_key_version: "encryptionkey@0"
I20260927 12:48:48.147709 14494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:48:48.160594 14494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:48:48.163717 14494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:48:48.163924 14494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:48:48.164515 14494 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:48:48.165308 14494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:48:48.165362 14494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:48:48.165413 14494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:48:48.165441 14494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:48:48.174011 14494 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37579
I20260927 12:48:48.174029 14622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37579 every 8 connection(s)
I20260927 12:48:48.175015 14494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:48:48.175447 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 14494
I20260927 12:48:48.175566 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:48:48.175836 392 external_mini_cluster.cc:1428] Setting key e2c6d60544eb6f08380b93adbfe646c4
Sep 27 12:48:48 dist-test-slave-lkn4 krb5kdc[14119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513328, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:48:48.188014 14203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:48.176925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57925 (local address 127.0.98.62:37309)
0927 12:48:48.177061 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 12:48:48.177065 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:48:48.178268 (+ 1203us) server_negotiation.cc:407] Connection header received
0927 12:48:48.179111 (+ 843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:48:48.179114 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:48:48.179167 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:48:48.179237 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:48:48.180375 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:48.180887 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:48.181988 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:48.182245 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:48.182331 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:48.184672 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:48:48.184677 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:48:48.184680 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:48:48.184706 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:48:48.186034 (+ 1328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:48:48.186531 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:48:48.186534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:48:48.186537 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:48:48.186612 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:48:48.186925 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:48:48.186928 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:48:48.186930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:48:48.187102 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:48:48.187192 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 12:48:48.187715 (+ 523us) server_negotiation.cc:299] Negotiation successful
0927 12:48:48.187838 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 12:48:48.188993 14625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:48.177420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37309 (local address 127.0.98.3:57925)
0927 12:48:48.178135 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:48.178183 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 12:48:48.178929 (+ 746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:48:48.179417 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:48:48.179425 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:48:48.179769 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:48:48.180244 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:48.180258 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:48.180995 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:48.180998 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:48.181875 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:48.181884 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:48.182376 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:48.182380 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:48.182543 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:48.183091 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:48:48.183110 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:48:48.184486 (+ 1376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:48:48.186173 (+ 1687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:48:48.186179 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:48:48.186196 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:48:48.186444 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:48:48.186699 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:48:48.186702 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:48:48.186704 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:48:48.186839 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:48:48.187220 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:48:48.187227 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:48:48.187471 (+ 244us) client_negotiation.cc:769] Sending connection context
0927 12:48:48.187657 (+ 186us) client_negotiation.cc:240] Negotiation successful
0927 12:48:48.187998 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":116,"threads_started":1}
I20260927 12:48:48.190110 14623 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37309
I20260927 12:48:48.190376 14623 heartbeater.cc:464] Registering TS with master...
I20260927 12:48:48.190865 14623 heartbeater.cc:511] Master 127.0.98.62:37309 requested a full tablet report, sending...
I20260927 12:48:48.191876 14164 ts_manager.cc:194] Registered new tserver with Master: 87c9851a903c4436b895d7da878979c7 (127.0.98.3:37579)
I20260927 12:48:48.192459 14164 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57925
I20260927 12:48:48.201459 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 12:48:48.210395 14203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:48.203362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45256 (local address 127.0.98.62:37309)
0927 12:48:48.203469 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 12:48:48.203473 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:48:48.203626 (+ 153us) server_negotiation.cc:407] Connection header received
0927 12:48:48.203768 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:48:48.203770 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:48:48.203815 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:48:48.203873 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:48:48.204502 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:48.205077 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:48.205866 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:48.206136 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:48.206218 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:48.207552 (+ 1334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:48:48.207555 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:48:48.207557 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:48:48.207593 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:48:48.208937 (+ 1344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:48:48.209316 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:48:48.209319 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:48:48.209320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:48:48.209407 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:48:48.209627 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:48:48.209630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:48:48.209631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:48:48.209870 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:48:48.209977 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 12:48:48.210172 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 12:48:48.210271 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 12:48:48.213234 14164 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45256:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:48:48.216434 14164 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:48:48.231252 14636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:48.224734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39543 (local address 127.0.0.1:40922)
0927 12:48:48.225596 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:48.225609 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:48:48.225691 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:48:48.226586 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:48:48.226589 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:48:48.226601 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:48:48.226718 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:48.226724 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:48.228164 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:48.228167 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:48.229774 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:48.229818 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:48.230934 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:48.230937 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:48.231023 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:48.231042 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 12:48:48.231099 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 12:48:48.231150 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":798,"mutex_wait_us":171,"thread_start_us":107,"threads_started":1}
I20260927 12:48:48.231560 14635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:48.224748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37656 (local address 127.0.98.1:37565)
0927 12:48:48.226123 (+ 1375us) server_negotiation.cc:206] Beginning negotiation
0927 12:48:48.226136 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 12:48:48.226156 (+ 20us) server_negotiation.cc:407] Connection header received
0927 12:48:48.226219 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:48:48.226223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:48:48.226368 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:48:48.226586 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:48:48.226781 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:48.227998 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:48.229910 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:48.230604 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:48.230698 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:48.230859 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 12:48:48.231329 (+ 470us) server_negotiation.cc:299] Negotiation successful
0927 12:48:48.231435 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1262,"thread_start_us":75,"threads_started":1}
I20260927 12:48:48.231868 14639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:48.225451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37016 (local address 127.0.98.3:37579)
0927 12:48:48.225893 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0927 12:48:48.225907 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 12:48:48.225931 (+ 24us) server_negotiation.cc:407] Connection header received
0927 12:48:48.226098 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:48:48.226103 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:48:48.226254 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:48:48.226392 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:48:48.228165 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:48.229087 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:48.230269 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:48.231020 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:48.231112 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:48.231234 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 12:48:48.231624 (+ 390us) server_negotiation.cc:299] Negotiation successful
0927 12:48:48.231723 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":115,"threads_started":1}
I20260927 12:48:48.231252 14634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:48.224617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37565 (local address 127.0.0.1:37656)
0927 12:48:48.225437 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:48.225490 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 12:48:48.225618 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:48:48.226502 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:48:48.226505 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:48:48.226521 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:48:48.226679 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:48.226684 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:48.228134 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:48.228138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:48.229774 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:48.229785 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:48.230711 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:48.230714 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:48.230818 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:48.230853 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 12:48:48.230957 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 12:48:48.231082 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"mutex_wait_us":130,"thread_start_us":76,"threads_started":1}
I20260927 12:48:48.232319 14638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:48.225217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40922 (local address 127.0.98.2:39543)
0927 12:48:48.225910 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0927 12:48:48.225921 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 12:48:48.225939 (+ 18us) server_negotiation.cc:407] Connection header received
0927 12:48:48.226013 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:48:48.226019 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:48:48.226272 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:48:48.226430 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:48:48.226842 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:48.227998 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:48.229928 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:48.230675 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:48.230774 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:48.230926 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 12:48:48.232106 (+ 1180us) server_negotiation.cc:299] Negotiation successful
0927 12:48:48.232188 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":132,"threads_started":1}
I20260927 12:48:48.231523 14637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:48.224941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37579 (local address 127.0.0.1:37016)
0927 12:48:48.225854 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:48.225866 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:48:48.225938 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:48:48.226392 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:48:48.226398 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:48:48.226419 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:48:48.226672 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:48.226680 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:48.229209 (+ 2529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:48.229212 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:48.230139 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:48.230149 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:48.231127 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:48.231130 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:48.231213 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:48.231234 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 12:48:48.231322 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 12:48:48.231389 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":858,"mutex_wait_us":36,"thread_start_us":91,"threads_started":1}
I20260927 12:48:48.234895 14423 tablet_service.cc:1467] Processing CreateTablet for tablet bfd9fae2544845e79e06072a4354e9e7 (DEFAULT_TABLE table=test-table [id=b54a56edc6024b96815caa6e945ec5d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:48:48.234918 14289 tablet_service.cc:1467] Processing CreateTablet for tablet bfd9fae2544845e79e06072a4354e9e7 (DEFAULT_TABLE table=test-table [id=b54a56edc6024b96815caa6e945ec5d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:48:48.234916 14557 tablet_service.cc:1467] Processing CreateTablet for tablet bfd9fae2544845e79e06072a4354e9e7 (DEFAULT_TABLE table=test-table [id=b54a56edc6024b96815caa6e945ec5d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:48:48.235916 14423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfd9fae2544845e79e06072a4354e9e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:48:48.235930 14289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfd9fae2544845e79e06072a4354e9e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:48:48.235937 14557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfd9fae2544845e79e06072a4354e9e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:48:48.242093 14640 tablet_bootstrap.cc:492] T bfd9fae2544845e79e06072a4354e9e7 P 8cbcb48cd2ab4a93956f5392df6c5983: Bootstrap starting.
I20260927 12:48:48.243489 14642 tablet_bootstrap.cc:492] T bfd9fae2544845e79e06072a4354e9e7 P 104cf2fdae024d4ebe3c93c3ea1afe37: Bootstrap starting.
I20260927 12:48:48.245548 14641 tablet_bootstrap.cc:492] T bfd9fae2544845e79e06072a4354e9e7 P 87c9851a903c4436b895d7da878979c7: Bootstrap starting.
I20260927 12:48:48.245599 14640 tablet_bootstrap.cc:654] T bfd9fae2544845e79e06072a4354e9e7 P 8cbcb48cd2ab4a93956f5392df6c5983: Neither blocks nor log segments found. Creating new log.
I20260927 12:48:48.245633 14642 tablet_bootstrap.cc:654] T bfd9fae2544845e79e06072a4354e9e7 P 104cf2fdae024d4ebe3c93c3ea1afe37: Neither blocks nor log segments found. Creating new log.
I20260927 12:48:48.246600 14640 log.cc:824] T bfd9fae2544845e79e06072a4354e9e7 P 8cbcb48cd2ab4a93956f5392df6c5983: Log is configured to *not* fsync() on all Append() calls
I20260927 12:48:48.246803 14642 log.cc:824] T bfd9fae2544845e79e06072a4354e9e7 P 104cf2fdae024d4ebe3c93c3ea1afe37: Log is configured to *not* fsync() on all Append() calls
I20260927 12:48:48.247498 14641 tablet_bootstrap.cc:654] T bfd9fae2544845e79e06072a4354e9e7 P 87c9851a903c4436b895d7da878979c7: Neither blocks nor log segments found. Creating new log.
I20260927 12:48:48.248428 14641 log.cc:824] T bfd9fae2544845e79e06072a4354e9e7 P 87c9851a903c4436b895d7da878979c7: Log is configured to *not* fsync() on all Append() calls
I20260927 12:48:48.248696 14640 tablet_bootstrap.cc:492] T bfd9fae2544845e79e06072a4354e9e7 P 8cbcb48cd2ab4a93956f5392df6c5983: No bootstrap required, opened a new log
I20260927 12:48:48.248696 14642 tablet_bootstrap.cc:492] T bfd9fae2544845e79e06072a4354e9e7 P 104cf2fdae024d4ebe3c93c3ea1afe37: No bootstrap required, opened a new log
I20260927 12:48:48.248883 14640 ts_tablet_manager.cc:1397] T bfd9fae2544845e79e06072a4354e9e7 P 8cbcb48cd2ab4a93956f5392df6c5983: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260927 12:48:48.248883 14642 ts_tablet_manager.cc:1397] T bfd9fae2544845e79e06072a4354e9e7 P 104cf2fdae024d4ebe3c93c3ea1afe37: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 12:48:48.250185 14641 tablet_bootstrap.cc:492] T bfd9fae2544845e79e06072a4354e9e7 P 87c9851a903c4436b895d7da878979c7: No bootstrap required, opened a new log
I20260927 12:48:48.250319 14641 ts_tablet_manager.cc:1397] T bfd9fae2544845e79e06072a4354e9e7 P 87c9851a903c4436b895d7da878979c7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 12:48:48.252985 14642 raft_consensus.cc:348] T bfd9fae2544845e79e06072a4354e9e7 P 104cf2fdae024d4ebe3c93c3ea1afe37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "104cf2fdae024d4ebe3c93c3ea1afe37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37565 } } peers { permanent_uuid: "8cbcb48cd2ab4a93956f5392df6c5983" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39543 } } peers { permanent_uuid: "87c9851a903c4436b895d7da878979c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37579 } }
I20260927 12:48:48.252985 14640 raft_consensus.cc:348] T bfd9fae2544845e79e06072a4354e9e7 P 8cbcb48cd2ab4a93956f5392df6c5983 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "104cf2fdae024d4ebe3c93c3ea1afe37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37565 } } peers { permanent_uuid: "8cbcb48cd2ab4a93956f5392df6c5983" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39543 } } peers { permanent_uuid: "87c9851a903c4436b895d7da878979c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37579 } }
I20260927 12:48:48.253275 14642 raft_consensus.cc:374] T bfd9fae2544845e79e06072a4354e9e7 P 104cf2fdae024d4ebe3c93c3ea1afe37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:48:48.253275 14640 raft_consensus.cc:374] T bfd9fae2544845e79e06072a4354e9e7 P 8cbcb48cd2ab4a93956f5392df6c5983 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:48:48.253336 14640 raft_consensus.cc:729] T bfd9fae2544845e79e06072a4354e9e7 P 8cbcb48cd2ab4a93956f5392df6c5983 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cbcb48cd2ab4a93956f5392df6c5983, State: Initialized, Role: FOLLOWER
I20260927 12:48:48.253362 14642 raft_consensus.cc:729] T bfd9fae2544845e79e06072a4354e9e7 P 104cf2fdae024d4ebe3c93c3ea1afe37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 104cf2fdae024d4ebe3c93c3ea1afe37, State: Initialized, Role: FOLLOWER
I20260927 12:48:48.253225 14641 raft_consensus.cc:348] T bfd9fae2544845e79e06072a4354e9e7 P 87c9851a903c4436b895d7da878979c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "104cf2fdae024d4ebe3c93c3ea1afe37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37565 } } peers { permanent_uuid: "8cbcb48cd2ab4a93956f5392df6c5983" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39543 } } peers { permanent_uuid: "87c9851a903c4436b895d7da878979c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37579 } }
I20260927 12:48:48.253487 14641 raft_consensus.cc:374] T bfd9fae2544845e79e06072a4354e9e7 P 87c9851a903c4436b895d7da878979c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:48:48.253545 14641 raft_consensus.cc:729] T bfd9fae2544845e79e06072a4354e9e7 P 87c9851a903c4436b895d7da878979c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87c9851a903c4436b895d7da878979c7, State: Initialized, Role: FOLLOWER
I20260927 12:48:48.253868 14640 consensus_queue.cc:260] T bfd9fae2544845e79e06072a4354e9e7 P 8cbcb48cd2ab4a93956f5392df6c5983 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "104cf2fdae024d4ebe3c93c3ea1afe37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37565 } } peers { permanent_uuid: "8cbcb48cd2ab4a93956f5392df6c5983" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39543 } } peers { permanent_uuid: "87c9851a903c4436b895d7da878979c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37579 } }
I20260927 12:48:48.253868 14642 consensus_queue.cc:260] T bfd9fae2544845e79e06072a4354e9e7 P 104cf2fdae024d4ebe3c93c3ea1afe37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "104cf2fdae024d4ebe3c93c3ea1afe37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37565 } } peers { permanent_uuid: "8cbcb48cd2ab4a93956f5392df6c5983" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39543 } } peers { permanent_uuid: "87c9851a903c4436b895d7da878979c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37579 } }
I20260927 12:48:48.253942 14641 consensus_queue.cc:260] T bfd9fae2544845e79e06072a4354e9e7 P 87c9851a903c4436b895d7da878979c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "104cf2fdae024d4ebe3c93c3ea1afe37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37565 } } peers { permanent_uuid: "8cbcb48cd2ab4a93956f5392df6c5983" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39543 } } peers { permanent_uuid: "87c9851a903c4436b895d7da878979c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37579 } }
I20260927 12:48:48.254797 14489 heartbeater.cc:503] Master 127.0.98.62:37309 was elected leader, sending a full tablet report...
I20260927 12:48:48.254807 14355 heartbeater.cc:503] Master 127.0.98.62:37309 was elected leader, sending a full tablet report...
I20260927 12:48:48.254909 14640 ts_tablet_manager.cc:1428] T bfd9fae2544845e79e06072a4354e9e7 P 8cbcb48cd2ab4a93956f5392df6c5983: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 12:48:48.254916 14641 ts_tablet_manager.cc:1428] T bfd9fae2544845e79e06072a4354e9e7 P 87c9851a903c4436b895d7da878979c7: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20260927 12:48:48.255203 14623 heartbeater.cc:503] Master 127.0.98.62:37309 was elected leader, sending a full tablet report...
I20260927 12:48:48.255499 14642 ts_tablet_manager.cc:1428] T bfd9fae2544845e79e06072a4354e9e7 P 104cf2fdae024d4ebe3c93c3ea1afe37: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260927 12:48:48.340255 14648 raft_consensus.cc:482] T bfd9fae2544845e79e06072a4354e9e7 P 87c9851a903c4436b895d7da878979c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:48:48.340451 14648 raft_consensus.cc:504] T bfd9fae2544845e79e06072a4354e9e7 P 87c9851a903c4436b895d7da878979c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "104cf2fdae024d4ebe3c93c3ea1afe37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37565 } } peers { permanent_uuid: "8cbcb48cd2ab4a93956f5392df6c5983" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39543 } } peers { permanent_uuid: "87c9851a903c4436b895d7da878979c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37579 } }
I20260927 12:48:48.341585 14648 leader_election.cc:288] T bfd9fae2544845e79e06072a4354e9e7 P 87c9851a903c4436b895d7da878979c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 104cf2fdae024d4ebe3c93c3ea1afe37 (127.0.98.1:37565), 8cbcb48cd2ab4a93956f5392df6c5983 (127.0.98.2:39543)
I20260927 12:48:48.345307 14635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:48.341917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50447 (local address 127.0.98.1:37565)
0927 12:48:48.342073 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 12:48:48.342077 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:48:48.342141 (+ 64us) server_negotiation.cc:407] Connection header received
0927 12:48:48.342294 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:48:48.342298 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:48:48.342351 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:48:48.342420 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:48:48.342683 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:48.343231 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:48.344344 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:48.344725 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:48.344804 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:48.344867 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 12:48:48.345106 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 12:48:48.345178 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 12:48:48.345315 14638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:48.341956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45137 (local address 127.0.98.2:39543)
0927 12:48:48.342084 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 12:48:48.342088 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:48:48.342331 (+ 243us) server_negotiation.cc:407] Connection header received
0927 12:48:48.342440 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:48:48.342443 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:48:48.342491 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:48:48.342567 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:48:48.342800 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:48.343263 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:48.344355 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:48.344752 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:48.344847 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:48.344906 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 12:48:48.345107 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 12:48:48.345179 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 12:48:48.345362 14649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:48.341855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39543 (local address 127.0.98.3:45137)
0927 12:48:48.342235 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:48.342248 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:48:48.342343 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:48:48.342579 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:48:48.342581 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:48:48.342594 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:48:48.342695 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:48.342701 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:48.343374 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:48.343376 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:48.344228 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:48.344236 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:48.344861 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:48.344864 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:48.344979 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:48.345016 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 12:48:48.345077 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 12:48:48.345178 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":43,"thread_start_us":157,"threads_started":1}
I20260927 12:48:48.345376 14625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:48.341839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37565 (local address 127.0.98.3:50447)
0927 12:48:48.342068 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:48.342083 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:48:48.342186 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:48:48.342437 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:48:48.342440 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:48:48.342457 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:48:48.342578 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:48.342585 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:48.343330 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:48.343333 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:48.344222 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:48.344231 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:48.344835 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:48.344839 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:48.344943 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:48.345012 (+ 69us) client_negotiation.cc:769] Sending connection context
0927 12:48:48.345075 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 12:48:48.345184 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":102}
I20260927 12:48:48.345968 14443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfd9fae2544845e79e06072a4354e9e7" candidate_uuid: "87c9851a903c4436b895d7da878979c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cbcb48cd2ab4a93956f5392df6c5983" is_pre_election: true
I20260927 12:48:48.346318 14443 raft_consensus.cc:2454] T bfd9fae2544845e79e06072a4354e9e7 P 8cbcb48cd2ab4a93956f5392df6c5983 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87c9851a903c4436b895d7da878979c7 in term 0.
I20260927 12:48:48.346524 14309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfd9fae2544845e79e06072a4354e9e7" candidate_uuid: "87c9851a903c4436b895d7da878979c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "104cf2fdae024d4ebe3c93c3ea1afe37" is_pre_election: true
I20260927 12:48:48.346863 14309 raft_consensus.cc:2454] T bfd9fae2544845e79e06072a4354e9e7 P 104cf2fdae024d4ebe3c93c3ea1afe37 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87c9851a903c4436b895d7da878979c7 in term 0.
I20260927 12:48:48.346827 14512 leader_election.cc:302] T bfd9fae2544845e79e06072a4354e9e7 P 87c9851a903c4436b895d7da878979c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87c9851a903c4436b895d7da878979c7, 8cbcb48cd2ab4a93956f5392df6c5983; no voters:
I20260927 12:48:48.347091 14648 raft_consensus.cc:2781] T bfd9fae2544845e79e06072a4354e9e7 P 87c9851a903c4436b895d7da878979c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:48:48.347196 14648 raft_consensus.cc:482] T bfd9fae2544845e79e06072a4354e9e7 P 87c9851a903c4436b895d7da878979c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:48:48.347254 14648 raft_consensus.cc:3037] T bfd9fae2544845e79e06072a4354e9e7 P 87c9851a903c4436b895d7da878979c7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:48:48.348347 14648 raft_consensus.cc:504] T bfd9fae2544845e79e06072a4354e9e7 P 87c9851a903c4436b895d7da878979c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "104cf2fdae024d4ebe3c93c3ea1afe37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37565 } } peers { permanent_uuid: "8cbcb48cd2ab4a93956f5392df6c5983" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39543 } } peers { permanent_uuid: "87c9851a903c4436b895d7da878979c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37579 } }
I20260927 12:48:48.348658 14648 leader_election.cc:288] T bfd9fae2544845e79e06072a4354e9e7 P 87c9851a903c4436b895d7da878979c7 [CANDIDATE]: Term 1 election: Requested vote from peers 104cf2fdae024d4ebe3c93c3ea1afe37 (127.0.98.1:37565), 8cbcb48cd2ab4a93956f5392df6c5983 (127.0.98.2:39543)
I20260927 12:48:48.348980 14309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfd9fae2544845e79e06072a4354e9e7" candidate_uuid: "87c9851a903c4436b895d7da878979c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "104cf2fdae024d4ebe3c93c3ea1afe37"
I20260927 12:48:48.349056 14443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfd9fae2544845e79e06072a4354e9e7" candidate_uuid: "87c9851a903c4436b895d7da878979c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cbcb48cd2ab4a93956f5392df6c5983"
I20260927 12:48:48.349100 14309 raft_consensus.cc:3037] T bfd9fae2544845e79e06072a4354e9e7 P 104cf2fdae024d4ebe3c93c3ea1afe37 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:48:48.349171 14443 raft_consensus.cc:3037] T bfd9fae2544845e79e06072a4354e9e7 P 8cbcb48cd2ab4a93956f5392df6c5983 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:48:48.350180 14443 raft_consensus.cc:2454] T bfd9fae2544845e79e06072a4354e9e7 P 8cbcb48cd2ab4a93956f5392df6c5983 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87c9851a903c4436b895d7da878979c7 in term 1.
I20260927 12:48:48.350193 14309 raft_consensus.cc:2454] T bfd9fae2544845e79e06072a4354e9e7 P 104cf2fdae024d4ebe3c93c3ea1afe37 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87c9851a903c4436b895d7da878979c7 in term 1.
I20260927 12:48:48.350422 14512 leader_election.cc:302] T bfd9fae2544845e79e06072a4354e9e7 P 87c9851a903c4436b895d7da878979c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87c9851a903c4436b895d7da878979c7, 8cbcb48cd2ab4a93956f5392df6c5983; no voters:
I20260927 12:48:48.350587 14648 raft_consensus.cc:2781] T bfd9fae2544845e79e06072a4354e9e7 P 87c9851a903c4436b895d7da878979c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:48:48.350790 14648 raft_consensus.cc:686] T bfd9fae2544845e79e06072a4354e9e7 P 87c9851a903c4436b895d7da878979c7 [term 1 LEADER]: Becoming Leader. State: Replica: 87c9851a903c4436b895d7da878979c7, State: Running, Role: LEADER
I20260927 12:48:48.351114 14648 consensus_queue.cc:237] T bfd9fae2544845e79e06072a4354e9e7 P 87c9851a903c4436b895d7da878979c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "104cf2fdae024d4ebe3c93c3ea1afe37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37565 } } peers { permanent_uuid: "8cbcb48cd2ab4a93956f5392df6c5983" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39543 } } peers { permanent_uuid: "87c9851a903c4436b895d7da878979c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37579 } }
I20260927 12:48:48.354473 14163 catalog_manager.cc:5495] T bfd9fae2544845e79e06072a4354e9e7 P 87c9851a903c4436b895d7da878979c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 87c9851a903c4436b895d7da878979c7 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "87c9851a903c4436b895d7da878979c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "104cf2fdae024d4ebe3c93c3ea1afe37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8cbcb48cd2ab4a93956f5392df6c5983" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87c9851a903c4436b895d7da878979c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37579 } health_report { overall_health: HEALTHY } } }
I20260927 12:48:48.386422 14639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:48.382728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37018 (local address 127.0.98.3:37579)
0927 12:48:48.382889 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 12:48:48.382893 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:48:48.382910 (+ 17us) server_negotiation.cc:407] Connection header received
0927 12:48:48.383018 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:48:48.383021 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:48:48.383095 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:48:48.383229 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:48:48.383574 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:48.384157 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:48.385030 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:48.385283 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:48.385360 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:48.385583 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:48:48.386024 (+ 441us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:48:48.386178 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 12:48:48.386204 (+ 26us) server_negotiation.cc:299] Negotiation successful
0927 12:48:48.386256 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 12:48:48.392205 14309 raft_consensus.cc:1261] T bfd9fae2544845e79e06072a4354e9e7 P 104cf2fdae024d4ebe3c93c3ea1afe37 [term 1 FOLLOWER]: Refusing update from remote peer 87c9851a903c4436b895d7da878979c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:48:48.392205 14443 raft_consensus.cc:1261] T bfd9fae2544845e79e06072a4354e9e7 P 8cbcb48cd2ab4a93956f5392df6c5983 [term 1 FOLLOWER]: Refusing update from remote peer 87c9851a903c4436b895d7da878979c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:48:48.392889 14648 consensus_queue.cc:1035] T bfd9fae2544845e79e06072a4354e9e7 P 87c9851a903c4436b895d7da878979c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "104cf2fdae024d4ebe3c93c3ea1afe37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:48:48.393071 14650 consensus_queue.cc:1035] T bfd9fae2544845e79e06072a4354e9e7 P 87c9851a903c4436b895d7da878979c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cbcb48cd2ab4a93956f5392df6c5983" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:48:48.401377 14655 mvcc.cc:205] Tried to move back new op lower bound from 7333942593087631360 to 7333942592929046528. Current Snapshot: MvccSnapshot[applied={T|T < 7333942593087631360}]
I20260927 12:48:48.407833 14162 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45256:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:48:48.407994 14162 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45256:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:48:48.410189 14162 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 76a78d2ebcd0446ebdafa41ff473c402: Sending DeleteTablet for 3 replicas of tablet bfd9fae2544845e79e06072a4354e9e7
I20260927 12:48:48.410899 14423 tablet_service.cc:1514] Processing DeleteTablet for tablet bfd9fae2544845e79e06072a4354e9e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:48:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40922
I20260927 12:48:48.410898 14289 tablet_service.cc:1514] Processing DeleteTablet for tablet bfd9fae2544845e79e06072a4354e9e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:48:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37656
I20260927 12:48:48.411113 14557 tablet_service.cc:1514] Processing DeleteTablet for tablet bfd9fae2544845e79e06072a4354e9e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:48:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37016
I20260927 12:48:48.411469 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 14226
I20260927 12:48:48.411487 14665 tablet_replica.cc:330] T bfd9fae2544845e79e06072a4354e9e7 P 87c9851a903c4436b895d7da878979c7: stopping tablet replica
I20260927 12:48:48.412653 14665 raft_consensus.cc:2229] T bfd9fae2544845e79e06072a4354e9e7 P 87c9851a903c4436b895d7da878979c7 [term 1 LEADER]: Raft consensus shutting down.
I20260927 12:48:48.413017 14665 raft_consensus.cc:2258] T bfd9fae2544845e79e06072a4354e9e7 P 87c9851a903c4436b895d7da878979c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:48:48.414074 14663 tablet_replica.cc:330] T bfd9fae2544845e79e06072a4354e9e7 P 8cbcb48cd2ab4a93956f5392df6c5983: stopping tablet replica
I20260927 12:48:48.414378 14663 raft_consensus.cc:2229] T bfd9fae2544845e79e06072a4354e9e7 P 8cbcb48cd2ab4a93956f5392df6c5983 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:48:48.414566 14665 ts_tablet_manager.cc:1905] T bfd9fae2544845e79e06072a4354e9e7 P 87c9851a903c4436b895d7da878979c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:48:48.414724 14663 raft_consensus.cc:2258] T bfd9fae2544845e79e06072a4354e9e7 P 8cbcb48cd2ab4a93956f5392df6c5983 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:48:48.416610 14663 ts_tablet_manager.cc:1905] T bfd9fae2544845e79e06072a4354e9e7 P 8cbcb48cd2ab4a93956f5392df6c5983: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:48:48.417348 14665 ts_tablet_manager.cc:1918] T bfd9fae2544845e79e06072a4354e9e7 P 87c9851a903c4436b895d7da878979c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:48:48.417505 14665 log.cc:1196] T bfd9fae2544845e79e06072a4354e9e7 P 87c9851a903c4436b895d7da878979c7: Deleting WAL directory at /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-2/wal/wals/bfd9fae2544845e79e06072a4354e9e7
I20260927 12:48:48.417886 14665 ts_tablet_manager.cc:1939] T bfd9fae2544845e79e06072a4354e9e7 P 87c9851a903c4436b895d7da878979c7: Deleting consensus metadata
I20260927 12:48:48.419251 14149 catalog_manager.cc:4841] TS 87c9851a903c4436b895d7da878979c7 (127.0.98.3:37579): tablet bfd9fae2544845e79e06072a4354e9e7 (table test-table [id=b54a56edc6024b96815caa6e945ec5d6]) successfully deleted
I20260927 12:48:48.419418 14663 ts_tablet_manager.cc:1918] T bfd9fae2544845e79e06072a4354e9e7 P 8cbcb48cd2ab4a93956f5392df6c5983: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:48:48.419505 14663 log.cc:1196] T bfd9fae2544845e79e06072a4354e9e7 P 8cbcb48cd2ab4a93956f5392df6c5983: Deleting WAL directory at /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-1/wal/wals/bfd9fae2544845e79e06072a4354e9e7
I20260927 12:48:48.420063 14663 ts_tablet_manager.cc:1939] T bfd9fae2544845e79e06072a4354e9e7 P 8cbcb48cd2ab4a93956f5392df6c5983: Deleting consensus metadata
W20260927 12:48:48.420183 14148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37565 (error 108)
I20260927 12:48:48.420446 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 14360
I20260927 12:48:48.422580 14637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:48.422240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37565 (local address 127.0.0.1:37658)
0927 12:48:48.422421 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:48.422492 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
W20260927 12:48:48.422765 14148 catalog_manager.cc:4584] TS 104cf2fdae024d4ebe3c93c3ea1afe37 (127.0.98.1:37565): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bfd9fae2544845e79e06072a4354e9e7: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37565: connect: Connection refused (error 111)
I20260927 12:48:48.426091 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 14494
I20260927 12:48:48.427196 14637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:48.426319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39543 (local address 127.0.0.1:40932)
0927 12:48:48.427065 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:48.427114 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:39543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":671}
I20260927 12:48:48.430974 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 14132
I20260927 12:48:48.436288 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37309
--webserver_interface=127.0.98.62
--webserver_port=44923
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37309
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:48:48.547098 14667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:48:48.547372 14667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:48:48.547413 14667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:48:48.551481 14667 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:48:48.551542 14667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:48:48.551560 14667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:48:48.551575 14667 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:48:48.551590 14667 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:48:48.556975 14667 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37309
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=44923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14667
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:48:48.558022 14667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:48:48.559103 14667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:48:48.571554 14674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:48:48.571539 14673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:48:48.571538 14676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:48:48.571883 14667 server_base.cc:1034] running on GCE node
I20260927 12:48:48.572470 14667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:48:48.573635 14667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:48:48.574806 14667 hybrid_clock.cc:648] HybridClock initialized: now 1790513328574774 us; error 45 us; skew 500 ppm
Sep 27 12:48:48 dist-test-slave-lkn4 krb5kdc[14119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513328, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:48:48.577637 14667 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:48:48.578639 14667 webserver.cc:460] Webserver started at http://127.0.98.62:44923/ using document root <none> and password file <none>
I20260927 12:48:48.579182 14667 fs_manager.cc:359] Metadata directory not provided
I20260927 12:48:48.579250 14667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:48:48.583076 14667 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260927 12:48:48.585127 14683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:48:48.586360 14667 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:48:48.586460 14667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/master-0/wal
uuid: "76a78d2ebcd0446ebdafa41ff473c402"
format_stamp: "Formatted at 2026-09-27 12:48:47 on dist-test-slave-lkn4"
server_key: "f7aafd8aa5d0a262908a480aa93c0a1f"
server_key_iv: "b340ad9cad868032c70ae03cb94d9054"
server_key_version: "encryptionkey@0"
I20260927 12:48:48.587023 14667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:48:48.598016 14667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:48:48.600750 14667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:48:48.600903 14667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:48:48.607569 14667 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37309
I20260927 12:48:48.607820 14735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37309 every 8 connection(s)
I20260927 12:48:48.608664 14667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:48:48.612033 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 14667
I20260927 12:48:48.612443 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:37565
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:37309
--webserver_port=43289
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 12:48:48.613011 14736 sys_catalog.cc:263] Verifying existing consensus state
I20260927 12:48:48.615144 14736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76a78d2ebcd0446ebdafa41ff473c402: Bootstrap starting.
I20260927 12:48:48.637547 14736 log.cc:824] T 00000000000000000000000000000000 P 76a78d2ebcd0446ebdafa41ff473c402: Log is configured to *not* fsync() on all Append() calls
I20260927 12:48:48.648084 14736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76a78d2ebcd0446ebdafa41ff473c402: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 12:48:48.648510 14736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76a78d2ebcd0446ebdafa41ff473c402: Bootstrap complete.
I20260927 12:48:48.652835 14736 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76a78d2ebcd0446ebdafa41ff473c402 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76a78d2ebcd0446ebdafa41ff473c402" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37309 } }
I20260927 12:48:48.653218 14736 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76a78d2ebcd0446ebdafa41ff473c402 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76a78d2ebcd0446ebdafa41ff473c402, State: Initialized, Role: FOLLOWER
I20260927 12:48:48.653717 14736 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76a78d2ebcd0446ebdafa41ff473c402 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76a78d2ebcd0446ebdafa41ff473c402" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37309 } }
I20260927 12:48:48.653834 14736 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76a78d2ebcd0446ebdafa41ff473c402 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:48:48.653900 14736 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76a78d2ebcd0446ebdafa41ff473c402 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:48:48.653988 14736 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76a78d2ebcd0446ebdafa41ff473c402 [term 1 FOLLOWER]: Advancing to term 2
I20260927 12:48:48.656283 14736 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76a78d2ebcd0446ebdafa41ff473c402 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76a78d2ebcd0446ebdafa41ff473c402" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37309 } }
I20260927 12:48:48.656620 14736 leader_election.cc:302] T 00000000000000000000000000000000 P 76a78d2ebcd0446ebdafa41ff473c402 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76a78d2ebcd0446ebdafa41ff473c402; no voters:
I20260927 12:48:48.656924 14736 leader_election.cc:288] T 00000000000000000000000000000000 P 76a78d2ebcd0446ebdafa41ff473c402 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 12:48:48.657027 14740 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76a78d2ebcd0446ebdafa41ff473c402 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 12:48:48.657225 14740 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76a78d2ebcd0446ebdafa41ff473c402 [term 2 LEADER]: Becoming Leader. State: Replica: 76a78d2ebcd0446ebdafa41ff473c402, State: Running, Role: LEADER
I20260927 12:48:48.657517 14740 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76a78d2ebcd0446ebdafa41ff473c402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76a78d2ebcd0446ebdafa41ff473c402" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37309 } }
I20260927 12:48:48.658187 14736 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76a78d2ebcd0446ebdafa41ff473c402 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:48:48.658491 14740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76a78d2ebcd0446ebdafa41ff473c402 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76a78d2ebcd0446ebdafa41ff473c402. Latest consensus state: current_term: 2 leader_uuid: "76a78d2ebcd0446ebdafa41ff473c402" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76a78d2ebcd0446ebdafa41ff473c402" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37309 } } }
I20260927 12:48:48.658614 14740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76a78d2ebcd0446ebdafa41ff473c402 [sys.catalog]: This master's current role is: LEADER
I20260927 12:48:48.658954 14744 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:48:48.659159 14741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76a78d2ebcd0446ebdafa41ff473c402 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "76a78d2ebcd0446ebdafa41ff473c402" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76a78d2ebcd0446ebdafa41ff473c402" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37309 } } }
I20260927 12:48:48.659257 14741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76a78d2ebcd0446ebdafa41ff473c402 [sys.catalog]: This master's current role is: LEADER
I20260927 12:48:48.663828 14744 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:48:48.664484 14744 catalog_manager.cc:1259] Loaded cluster ID: 6c455548fd1a40088a72a562cfb393ba
I20260927 12:48:48.664534 14744 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:48:48.665608 14744 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260927 12:48:48.668514 14744 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260927 12:48:48.672596 14744 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:48:48.673264 14744 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 76a78d2ebcd0446ebdafa41ff473c402: Loaded TSK: 0
I20260927 12:48:48.674230 14744 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 12:48:48.750728 14738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:48:48.751008 14738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:48:48.751068 14738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:48:48.755105 14738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:48:48.755169 14738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:48:48.755247 14738 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:48:48.760697 14738 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:37565
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=43289
--tserver_master_addrs=127.0.98.62:37309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:48:48.761672 14738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:48:48.762658 14738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:48:48.771667 14763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:48:48.771718 14766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:48:48.771677 14764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:48:48.772164 14738 server_base.cc:1034] running on GCE node
I20260927 12:48:48.772596 14738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:48:48.773345 14738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:48:48.774534 14738 hybrid_clock.cc:648] HybridClock initialized: now 1790513328774510 us; error 30 us; skew 500 ppm
Sep 27 12:48:48 dist-test-slave-lkn4 krb5kdc[14119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513328, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:48:48.777742 14738 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 12:48:48.778887 14738 webserver.cc:460] Webserver started at http://127.0.98.1:43289/ using document root <none> and password file <none>
I20260927 12:48:48.779608 14738 fs_manager.cc:359] Metadata directory not provided
I20260927 12:48:48.779721 14738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:48:48.784964 14738 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 12:48:48.787850 14773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:48:48.789402 14738 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 12:48:48.789536 14738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-0/wal
uuid: "104cf2fdae024d4ebe3c93c3ea1afe37"
format_stamp: "Formatted at 2026-09-27 12:48:47 on dist-test-slave-lkn4"
server_key: "7999efb846f0c33cee2dd92b36225e5d"
server_key_iv: "25a33bbfb862beab983e0ff1d9a486a5"
server_key_version: "encryptionkey@0"
I20260927 12:48:48.790299 14738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:48:48.818988 14738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:48:48.822409 14738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:48:48.822599 14738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:48:48.823251 14738 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:48:48.824476 14780 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 12:48:48.828614 14738 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 12:48:48.828694 14738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260927 12:48:48.828766 14738 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 12:48:48.830597 14738 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 12:48:48.830641 14738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260927 12:48:48.830929 14780 tablet_bootstrap.cc:492] T bfd9fae2544845e79e06072a4354e9e7 P 104cf2fdae024d4ebe3c93c3ea1afe37: Bootstrap starting.
I20260927 12:48:48.840499 14738 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37565
I20260927 12:48:48.840519 14887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37565 every 8 connection(s)
I20260927 12:48:48.841712 14738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:48:48.849696 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 14738
I20260927 12:48:48.849778 14780 log.cc:824] T bfd9fae2544845e79e06072a4354e9e7 P 104cf2fdae024d4ebe3c93c3ea1afe37: Log is configured to *not* fsync() on all Append() calls
I20260927 12:48:48.850162 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:39543
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:37309
--webserver_port=46479
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 12:48:48 dist-test-slave-lkn4 krb5kdc[14119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513328, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:48:48.858227 14780 tablet_bootstrap.cc:492] T bfd9fae2544845e79e06072a4354e9e7 P 104cf2fdae024d4ebe3c93c3ea1afe37: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 12:48:48.858688 14890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:48.843909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60005 (local address 127.0.98.62:37309)
0927 12:48:48.844258 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0927 12:48:48.844278 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 12:48:48.844611 (+ 333us) server_negotiation.cc:407] Connection header received
0927 12:48:48.845733 (+ 1122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:48:48.845743 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:48:48.846091 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:48:48.846402 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:48:48.847510 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:48.848460 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:48.849683 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:48.850404 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:48.850590 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:48.853882 (+ 3292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:48:48.853887 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:48:48.853901 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:48:48.853929 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:48:48.855582 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:48:48.856167 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:48:48.856173 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:48:48.856178 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:48:48.856297 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:48:48.856606 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:48:48.856610 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:48:48.856612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:48:48.856944 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:48:48.857054 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 12:48:48.857584 (+ 530us) server_negotiation.cc:299] Negotiation successful
0927 12:48:48.857843 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":107,"threads_started":1}
I20260927 12:48:48.858702 14891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:48.843930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37309 (local address 127.0.98.1:60005)
0927 12:48:48.844459 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:48.844501 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 12:48:48.845332 (+ 831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:48:48.846471 (+ 1139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:48:48.846479 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:48:48.846890 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:48:48.847353 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:48.847367 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:48.848603 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:48.848605 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:48.849546 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:48.849557 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:48.850574 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:48.850578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:48.850765 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:48.851399 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:48:48.851419 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:48:48.853640 (+ 2221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:48:48.855693 (+ 2053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:48:48.855698 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:48:48.855711 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:48:48.855958 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:48:48.856394 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:48:48.856398 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:48:48.856399 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:48:48.856525 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:48:48.857057 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:48:48.857064 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:48:48.857301 (+ 237us) client_negotiation.cc:769] Sending connection context
0927 12:48:48.857456 (+ 155us) client_negotiation.cc:240] Negotiation successful
0927 12:48:48.857774 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":167,"threads_started":1}
I20260927 12:48:48.858716 14780 tablet_bootstrap.cc:492] T bfd9fae2544845e79e06072a4354e9e7 P 104cf2fdae024d4ebe3c93c3ea1afe37: Bootstrap complete.
I20260927 12:48:48.860260 14780 ts_tablet_manager.cc:1397] T bfd9fae2544845e79e06072a4354e9e7 P 104cf2fdae024d4ebe3c93c3ea1afe37: Time spent bootstrapping tablet: real 0.030s user 0.024s sys 0.004s
I20260927 12:48:48.861389 14888 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37309
I20260927 12:48:48.861622 14888 heartbeater.cc:464] Registering TS with master...
I20260927 12:48:48.862151 14888 heartbeater.cc:511] Master 127.0.98.62:37309 requested a full tablet report, sending...
I20260927 12:48:48.862695 14780 raft_consensus.cc:348] T bfd9fae2544845e79e06072a4354e9e7 P 104cf2fdae024d4ebe3c93c3ea1afe37 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "104cf2fdae024d4ebe3c93c3ea1afe37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37565 } } peers { permanent_uuid: "8cbcb48cd2ab4a93956f5392df6c5983" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39543 } } peers { permanent_uuid: "87c9851a903c4436b895d7da878979c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37579 } }
I20260927 12:48:48.863086 14780 raft_consensus.cc:729] T bfd9fae2544845e79e06072a4354e9e7 P 104cf2fdae024d4ebe3c93c3ea1afe37 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 104cf2fdae024d4ebe3c93c3ea1afe37, State: Initialized, Role: FOLLOWER
I20260927 12:48:48.863529 14780 consensus_queue.cc:260] T bfd9fae2544845e79e06072a4354e9e7 P 104cf2fdae024d4ebe3c93c3ea1afe37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "104cf2fdae024d4ebe3c93c3ea1afe37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37565 } } peers { permanent_uuid: "8cbcb48cd2ab4a93956f5392df6c5983" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39543 } } peers { permanent_uuid: "87c9851a903c4436b895d7da878979c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37579 } }
I20260927 12:48:48.864506 14780 ts_tablet_manager.cc:1428] T bfd9fae2544845e79e06072a4354e9e7 P 104cf2fdae024d4ebe3c93c3ea1afe37: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260927 12:48:48.864702 14700 ts_manager.cc:194] Registered new tserver with Master: 104cf2fdae024d4ebe3c93c3ea1afe37 (127.0.98.1:37565)
I20260927 12:48:48.868332 14700 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60005
I20260927 12:48:48.871176 14888 heartbeater.cc:503] Master 127.0.98.62:37309 was elected leader, sending a full tablet report...
Sep 27 12:48:48 dist-test-slave-lkn4 krb5kdc[14119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513328, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 12:48:48.877350 14896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:48.867634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37672 (local address 127.0.98.1:37565)
0927 12:48:48.868239 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0927 12:48:48.868251 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 12:48:48.868273 (+ 22us) server_negotiation.cc:407] Connection header received
0927 12:48:48.868334 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:48:48.868339 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:48:48.868519 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:48:48.868645 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:48:48.869428 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:48.870232 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:48.871091 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:48.871492 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:48.871608 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:48.874062 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:48:48.874074 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:48:48.874087 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:48:48.874112 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:48:48.875626 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:48:48.876105 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:48:48.876109 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:48:48.876110 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:48:48.876180 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:48:48.876413 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:48:48.876415 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:48:48.876416 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:48:48.876631 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:48:48.876722 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 12:48:48.877076 (+ 354us) server_negotiation.cc:299] Negotiation successful
0927 12:48:48.877201 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":515,"thread_start_us":135,"threads_started":1}
I20260927 12:48:48.877380 14895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:48.867485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37565 (local address 127.0.0.1:37672)
0927 12:48:48.867938 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:48.867987 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 12:48:48.868118 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:48:48.868740 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:48:48.868746 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:48:48.869181 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:48:48.869320 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:48.869330 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:48.870338 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:48.870342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:48.870974 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:48.870984 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:48.871602 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:48.871620 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:48.871737 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:48.872329 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:48:48.872343 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:48:48.873890 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:48:48.875726 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:48:48.875734 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:48:48.875739 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:48:48.876011 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:48:48.876249 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:48:48.876252 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:48:48.876254 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:48:48.876341 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:48:48.876722 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:48:48.876727 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:48:48.876881 (+ 154us) client_negotiation.cc:769] Sending connection context
0927 12:48:48.877019 (+ 138us) client_negotiation.cc:240] Negotiation successful
0927 12:48:48.877208 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":130,"threads_started":1}
I20260927 12:48:48.878347 14822 tablet_service.cc:1514] Processing DeleteTablet for tablet bfd9fae2544845e79e06072a4354e9e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:48:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37672
I20260927 12:48:48.879191 14897 tablet_replica.cc:330] stopping tablet replica
I20260927 12:48:48.879551 14897 raft_consensus.cc:2229] T bfd9fae2544845e79e06072a4354e9e7 P 104cf2fdae024d4ebe3c93c3ea1afe37 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:48:48.879818 14897 raft_consensus.cc:2258] T bfd9fae2544845e79e06072a4354e9e7 P 104cf2fdae024d4ebe3c93c3ea1afe37 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:48:48.880877 14897 ts_tablet_manager.cc:1905] T bfd9fae2544845e79e06072a4354e9e7 P 104cf2fdae024d4ebe3c93c3ea1afe37: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:48:48.885295 14897 ts_tablet_manager.cc:1918] T bfd9fae2544845e79e06072a4354e9e7 P 104cf2fdae024d4ebe3c93c3ea1afe37: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:48:48.885399 14897 log.cc:1196] T bfd9fae2544845e79e06072a4354e9e7 P 104cf2fdae024d4ebe3c93c3ea1afe37: Deleting WAL directory at /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-0/wal/wals/bfd9fae2544845e79e06072a4354e9e7
I20260927 12:48:48.885658 14897 ts_tablet_manager.cc:1939] T bfd9fae2544845e79e06072a4354e9e7 P 104cf2fdae024d4ebe3c93c3ea1afe37: Deleting consensus metadata
I20260927 12:48:48.887876 14684 catalog_manager.cc:4841] TS 104cf2fdae024d4ebe3c93c3ea1afe37 (127.0.98.1:37565): tablet bfd9fae2544845e79e06072a4354e9e7 (table test-table [id=b54a56edc6024b96815caa6e945ec5d6]) successfully deleted
W20260927 12:48:48.968485 14893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:48:48.968729 14893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:48:48.968772 14893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:48:48.972808 14893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:48:48.972874 14893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:48:48.972934 14893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:48:48.978189 14893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:39543
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=46479
--tserver_master_addrs=127.0.98.62:37309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:48:48.979249 14893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:48:48.980212 14893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:48:48.989925 14903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:48:48.989951 14902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:48:48.990008 14905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:48:48.990126 14893 server_base.cc:1034] running on GCE node
I20260927 12:48:48.990658 14893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:48:48.991328 14893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:48:48.992496 14893 hybrid_clock.cc:648] HybridClock initialized: now 1790513328992466 us; error 43 us; skew 500 ppm
Sep 27 12:48:48 dist-test-slave-lkn4 krb5kdc[14119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513328, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:48:48.995383 14893 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 12:48:48.996249 14893 webserver.cc:460] Webserver started at http://127.0.98.2:46479/ using document root <none> and password file <none>
I20260927 12:48:48.996768 14893 fs_manager.cc:359] Metadata directory not provided
I20260927 12:48:48.996836 14893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:48:49.000739 14893 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260927 12:48:49.002744 14912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:48:49.003962 14893 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:48:49.004055 14893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-1/wal
uuid: "8cbcb48cd2ab4a93956f5392df6c5983"
format_stamp: "Formatted at 2026-09-27 12:48:47 on dist-test-slave-lkn4"
server_key: "6050048a83049ae6e551f24e3f21eab9"
server_key_iv: "d51ae3f054663ff1319b0f5dec057945"
server_key_version: "encryptionkey@0"
I20260927 12:48:49.004551 14893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:48:49.027508 14893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:48:49.030505 14893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:48:49.030660 14893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:48:49.031203 14893 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:48:49.031998 14893 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:48:49.032052 14893 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:48:49.032094 14893 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:48:49.032114 14893 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:48:49.040414 14893 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39543
I20260927 12:48:49.040437 15025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39543 every 8 connection(s)
I20260927 12:48:49.041486 14893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:48:49.047465 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 14893
I20260927 12:48:49.047843 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:37579
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:37309
--webserver_port=35697
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 12:48:49 dist-test-slave-lkn4 krb5kdc[14119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513328, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:48:49.054817 14890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:49.043254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60201 (local address 127.0.98.62:37309)
0927 12:48:49.043382 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 12:48:49.043385 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:48:49.044397 (+ 1012us) server_negotiation.cc:407] Connection header received
0927 12:48:49.045299 (+ 902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:48:49.045303 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:48:49.045364 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:48:49.045436 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:48:49.046671 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:49.047206 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:49.048443 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:49.048666 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:49.048754 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:49.051400 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:48:49.051403 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:48:49.051405 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:48:49.051434 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:48:49.052776 (+ 1342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:48:49.053254 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:48:49.053257 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:48:49.053258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:48:49.053354 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:48:49.053657 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:48:49.053661 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:48:49.053662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:48:49.053841 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:48:49.053926 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 12:48:49.054509 (+ 583us) server_negotiation.cc:299] Negotiation successful
0927 12:48:49.054647 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 12:48:49.055660 15028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:49.043622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37309 (local address 127.0.98.2:60201)
0927 12:48:49.044242 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:49.044282 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 12:48:49.045084 (+ 802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:48:49.045620 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:48:49.045628 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:48:49.045993 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:48:49.046519 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:49.046534 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:49.047326 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:49.047329 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:49.048327 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:49.048337 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:49.048814 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:49.048818 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:49.048982 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:49.049573 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:48:49.049588 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:48:49.051230 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:48:49.052887 (+ 1657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:48:49.052893 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:48:49.052910 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:48:49.053162 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:48:49.053435 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:48:49.053438 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:48:49.053440 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:48:49.053575 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:48:49.053937 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:48:49.053945 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:48:49.054277 (+ 332us) client_negotiation.cc:769] Sending connection context
0927 12:48:49.054452 (+ 175us) client_negotiation.cc:240] Negotiation successful
0927 12:48:49.054802 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":98,"threads_started":1}
I20260927 12:48:49.057756 15026 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37309
I20260927 12:48:49.058063 15026 heartbeater.cc:464] Registering TS with master...
I20260927 12:48:49.058624 15026 heartbeater.cc:511] Master 127.0.98.62:37309 requested a full tablet report, sending...
I20260927 12:48:49.059718 14700 ts_manager.cc:194] Registered new tserver with Master: 8cbcb48cd2ab4a93956f5392df6c5983 (127.0.98.2:39543)
I20260927 12:48:49.060467 14700 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60201
W20260927 12:48:49.157521 15029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:48:49.157800 15029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:48:49.157850 15029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:48:49.162026 15029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:48:49.162119 15029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:48:49.162178 15029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:48:49.167613 15029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:37579
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=35697
--tserver_master_addrs=127.0.98.62:37309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:48:49.168955 15029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:48:49.169973 15029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:48:49.179823 15034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:48:49.179951 15029 server_base.cc:1034] running on GCE node
W20260927 12:48:49.179796 15035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:48:49.179776 15037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:48:49.180455 15029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:48:49.181056 15029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:48:49.182247 15029 hybrid_clock.cc:648] HybridClock initialized: now 1790513329182231 us; error 25 us; skew 500 ppm
Sep 27 12:48:49 dist-test-slave-lkn4 krb5kdc[14119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513329, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:48:49.184790 15029 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 12:48:49.185600 15029 webserver.cc:460] Webserver started at http://127.0.98.3:35697/ using document root <none> and password file <none>
I20260927 12:48:49.186158 15029 fs_manager.cc:359] Metadata directory not provided
I20260927 12:48:49.186244 15029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:48:49.189920 15029 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:48:49.191946 15044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:48:49.193167 15029 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:48:49.193324 15029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-2/wal
uuid: "87c9851a903c4436b895d7da878979c7"
format_stamp: "Formatted at 2026-09-27 12:48:48 on dist-test-slave-lkn4"
server_key: "c8ecfc2f6ec145221221b98795cc6cee"
server_key_iv: "424022327d0b2b00281f1069faf133aa"
server_key_version: "encryptionkey@0"
I20260927 12:48:49.193866 15029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:48:49.205087 15029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:48:49.208063 15029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:48:49.208253 15029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:48:49.208832 15029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:48:49.209636 15029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:48:49.209683 15029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:48:49.209745 15029 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:48:49.209771 15029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:48:49.217993 15029 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37579
I20260927 12:48:49.218014 15157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37579 every 8 connection(s)
I20260927 12:48:49.218961 15029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:48:49.224023 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 15029
Sep 27 12:48:49 dist-test-slave-lkn4 krb5kdc[14119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513329, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:48:49.232096 15162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:49.224615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45262 (local address 127.0.98.62:37309)
0927 12:48:49.224842 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 12:48:49.224845 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:48:49.224971 (+ 126us) server_negotiation.cc:407] Connection header received
0927 12:48:49.225094 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:48:49.225096 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:48:49.225130 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:48:49.225198 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:48:49.226012 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:49.226541 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:49.227494 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:49.227711 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:49.227794 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:49.229135 (+ 1341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:48:49.229139 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:48:49.229140 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:48:49.229162 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:48:49.230603 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:48:49.230982 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:48:49.230986 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:48:49.230987 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:48:49.231057 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:48:49.231287 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:48:49.231290 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:48:49.231292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:48:49.231479 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:48:49.231677 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0927 12:48:49.231810 (+ 133us) server_negotiation.cc:299] Negotiation successful
0927 12:48:49.231928 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":75,"threads_started":1}
I20260927 12:48:49.232613 14890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:49.220635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42023 (local address 127.0.98.62:37309)
0927 12:48:49.220777 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 12:48:49.220781 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:48:49.221847 (+ 1066us) server_negotiation.cc:407] Connection header received
0927 12:48:49.222659 (+ 812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:48:49.222661 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:48:49.222707 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:48:49.222793 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:48:49.223927 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:49.224528 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:49.226246 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:49.226469 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:49.226554 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:49.229174 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:48:49.229177 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:48:49.229179 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:48:49.229213 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:48:49.230683 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:48:49.231170 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:48:49.231173 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:48:49.231175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:48:49.231253 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:48:49.231564 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:48:49.231568 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:48:49.231569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:48:49.231729 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:48:49.231812 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 12:48:49.232382 (+ 570us) server_negotiation.cc:299] Negotiation successful
0927 12:48:49.232490 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":50}
I20260927 12:48:49.233624 15160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:49.221107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37309 (local address 127.0.98.3:42023)
0927 12:48:49.221707 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:49.221745 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 12:48:49.222485 (+ 740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:48:49.222963 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:48:49.222969 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:48:49.223306 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:48:49.223768 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:49.223786 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:49.224658 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:49.224660 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:49.226133 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:49.226143 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:49.226604 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:49.226608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:49.226789 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:49.227363 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:48:49.227379 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:48:49.229009 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:48:49.230795 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:48:49.230802 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:48:49.230819 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:48:49.231075 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:48:49.231334 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:48:49.231337 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:48:49.231339 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:48:49.231479 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:48:49.231824 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:48:49.231831 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:48:49.232113 (+ 282us) client_negotiation.cc:769] Sending connection context
0927 12:48:49.232327 (+ 214us) client_negotiation.cc:240] Negotiation successful
0927 12:48:49.232684 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":109,"threads_started":1}
I20260927 12:48:49.234943 15158 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37309
I20260927 12:48:49.235208 15158 heartbeater.cc:464] Registering TS with master...
I20260927 12:48:49.235803 15158 heartbeater.cc:511] Master 127.0.98.62:37309 requested a full tablet report, sending...
I20260927 12:48:49.236948 14700 ts_manager.cc:194] Registered new tserver with Master: 87c9851a903c4436b895d7da878979c7 (127.0.98.3:37579)
I20260927 12:48:49.237710 14700 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42023
I20260927 12:48:49.245477 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 12:48:49.249356 14890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:49.246028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45266 (local address 127.0.98.62:37309)
0927 12:48:49.246175 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 12:48:49.246179 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:48:49.246350 (+ 171us) server_negotiation.cc:407] Connection header received
0927 12:48:49.246485 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:48:49.246487 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:48:49.246586 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:48:49.246667 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:48:49.246866 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:49.247350 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:49.248190 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:49.248390 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:49.248451 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:49.248663 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:48:49.249054 (+ 391us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:48:49.249135 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 12:48:49.249179 (+ 44us) server_negotiation.cc:299] Negotiation successful
0927 12:48:49.249223 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 12:48:49.249972 14700 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:45266:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:48:49.253193 14700 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:48:49.261369 14822 tablet_service.cc:1467] Processing CreateTablet for tablet d2d209f53f204d5288490f1c35807422 (DEFAULT_TABLE table=test-table [id=30d7b1b4b6074e5f80a6fce87c95f341]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:48:49.262110 14822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2d209f53f204d5288490f1c35807422. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:48:49.265017 14780 tablet_bootstrap.cc:492] T d2d209f53f204d5288490f1c35807422 P 104cf2fdae024d4ebe3c93c3ea1afe37: Bootstrap starting.
I20260927 12:48:49.265094 14895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:49.260092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37579 (local address 127.0.0.1:37030)
0927 12:48:49.260266 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:49.260282 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:48:49.260366 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:48:49.261305 (+ 939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:48:49.261308 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:48:49.261326 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:48:49.261430 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:49.261436 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:49.262881 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:49.262885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:49.263885 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:49.263894 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:49.264695 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:49.264699 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:49.264798 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:49.264841 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 12:48:49.264882 (+ 41us) client_negotiation.cc:240] Negotiation successful
0927 12:48:49.264962 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102}
I20260927 12:48:49.265799 15167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:49.260200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37030 (local address 127.0.98.3:37579)
0927 12:48:49.260697 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0927 12:48:49.260712 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 12:48:49.260738 (+ 26us) server_negotiation.cc:407] Connection header received
0927 12:48:49.260826 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:48:49.260834 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:48:49.261095 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:48:49.261340 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:48:49.261541 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:49.262738 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:49.263995 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:49.264563 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:49.265363 (+ 800us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:49.265521 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 12:48:49.265573 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 12:48:49.265666 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":137,"threads_started":1}
I20260927 12:48:49.265897 14780 tablet_bootstrap.cc:654] T d2d209f53f204d5288490f1c35807422 P 104cf2fdae024d4ebe3c93c3ea1afe37: Neither blocks nor log segments found. Creating new log.
I20260927 12:48:49.266153 15168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:49.260495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39543 (local address 127.0.0.1:40944)
0927 12:48:49.260754 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:49.260765 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 12:48:49.260838 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:48:49.261697 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:48:49.261700 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:48:49.261716 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:48:49.261802 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:49.261809 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:49.263483 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:49.263487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:49.264641 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:49.264655 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:49.265687 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:49.265691 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:49.265785 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:49.265811 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 12:48:49.265894 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 12:48:49.265971 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":113,"threads_started":1}
I20260927 12:48:49.266232 15169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:49.260965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40944 (local address 127.0.98.2:39543)
0927 12:48:49.261229 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0927 12:48:49.261240 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 12:48:49.261261 (+ 21us) server_negotiation.cc:407] Connection header received
0927 12:48:49.261349 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:48:49.261354 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:48:49.261562 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:48:49.262074 (+ 512us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:48:49.262243 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:49.263331 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:49.264813 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:49.265549 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:49.265674 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:49.265816 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 12:48:49.265969 (+ 153us) server_negotiation.cc:299] Negotiation successful
0927 12:48:49.266091 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":88,"threads_started":1}
I20260927 12:48:49.267402 14780 tablet_bootstrap.cc:492] T d2d209f53f204d5288490f1c35807422 P 104cf2fdae024d4ebe3c93c3ea1afe37: No bootstrap required, opened a new log
I20260927 12:48:49.267493 14780 ts_tablet_manager.cc:1397] T d2d209f53f204d5288490f1c35807422 P 104cf2fdae024d4ebe3c93c3ea1afe37: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260927 12:48:49.267925 14780 raft_consensus.cc:348] T d2d209f53f204d5288490f1c35807422 P 104cf2fdae024d4ebe3c93c3ea1afe37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c9851a903c4436b895d7da878979c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37579 } } peers { permanent_uuid: "8cbcb48cd2ab4a93956f5392df6c5983" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39543 } } peers { permanent_uuid: "104cf2fdae024d4ebe3c93c3ea1afe37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37565 } }
I20260927 12:48:49.268043 14780 raft_consensus.cc:374] T d2d209f53f204d5288490f1c35807422 P 104cf2fdae024d4ebe3c93c3ea1afe37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:48:49.268078 14780 raft_consensus.cc:729] T d2d209f53f204d5288490f1c35807422 P 104cf2fdae024d4ebe3c93c3ea1afe37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 104cf2fdae024d4ebe3c93c3ea1afe37, State: Initialized, Role: FOLLOWER
I20260927 12:48:49.268203 14780 consensus_queue.cc:260] T d2d209f53f204d5288490f1c35807422 P 104cf2fdae024d4ebe3c93c3ea1afe37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c9851a903c4436b895d7da878979c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37579 } } peers { permanent_uuid: "8cbcb48cd2ab4a93956f5392df6c5983" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39543 } } peers { permanent_uuid: "104cf2fdae024d4ebe3c93c3ea1afe37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37565 } }
I20260927 12:48:49.268445 14780 ts_tablet_manager.cc:1428] T d2d209f53f204d5288490f1c35807422 P 104cf2fdae024d4ebe3c93c3ea1afe37: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 12:48:49.269470 14960 tablet_service.cc:1467] Processing CreateTablet for tablet d2d209f53f204d5288490f1c35807422 (DEFAULT_TABLE table=test-table [id=30d7b1b4b6074e5f80a6fce87c95f341]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:48:49.270535 14960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2d209f53f204d5288490f1c35807422. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:48:49.275832 15092 tablet_service.cc:1467] Processing CreateTablet for tablet d2d209f53f204d5288490f1c35807422 (DEFAULT_TABLE table=test-table [id=30d7b1b4b6074e5f80a6fce87c95f341]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:48:49.276690 15092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2d209f53f204d5288490f1c35807422. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:48:49.279774 15171 tablet_bootstrap.cc:492] T d2d209f53f204d5288490f1c35807422 P 8cbcb48cd2ab4a93956f5392df6c5983: Bootstrap starting.
I20260927 12:48:49.282301 15171 tablet_bootstrap.cc:654] T d2d209f53f204d5288490f1c35807422 P 8cbcb48cd2ab4a93956f5392df6c5983: Neither blocks nor log segments found. Creating new log.
I20260927 12:48:49.283034 15171 log.cc:824] T d2d209f53f204d5288490f1c35807422 P 8cbcb48cd2ab4a93956f5392df6c5983: Log is configured to *not* fsync() on all Append() calls
I20260927 12:48:49.283135 15172 tablet_bootstrap.cc:492] T d2d209f53f204d5288490f1c35807422 P 87c9851a903c4436b895d7da878979c7: Bootstrap starting.
I20260927 12:48:49.284881 15171 tablet_bootstrap.cc:492] T d2d209f53f204d5288490f1c35807422 P 8cbcb48cd2ab4a93956f5392df6c5983: No bootstrap required, opened a new log
I20260927 12:48:49.284925 15172 tablet_bootstrap.cc:654] T d2d209f53f204d5288490f1c35807422 P 87c9851a903c4436b895d7da878979c7: Neither blocks nor log segments found. Creating new log.
I20260927 12:48:49.285071 15171 ts_tablet_manager.cc:1397] T d2d209f53f204d5288490f1c35807422 P 8cbcb48cd2ab4a93956f5392df6c5983: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 12:48:49.285903 15172 log.cc:824] T d2d209f53f204d5288490f1c35807422 P 87c9851a903c4436b895d7da878979c7: Log is configured to *not* fsync() on all Append() calls
I20260927 12:48:49.287940 15172 tablet_bootstrap.cc:492] T d2d209f53f204d5288490f1c35807422 P 87c9851a903c4436b895d7da878979c7: No bootstrap required, opened a new log
I20260927 12:48:49.288146 15172 ts_tablet_manager.cc:1397] T d2d209f53f204d5288490f1c35807422 P 87c9851a903c4436b895d7da878979c7: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 12:48:49.288774 15171 raft_consensus.cc:348] T d2d209f53f204d5288490f1c35807422 P 8cbcb48cd2ab4a93956f5392df6c5983 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c9851a903c4436b895d7da878979c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37579 } } peers { permanent_uuid: "8cbcb48cd2ab4a93956f5392df6c5983" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39543 } } peers { permanent_uuid: "104cf2fdae024d4ebe3c93c3ea1afe37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37565 } }
I20260927 12:48:49.289088 15171 raft_consensus.cc:374] T d2d209f53f204d5288490f1c35807422 P 8cbcb48cd2ab4a93956f5392df6c5983 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:48:49.289166 15171 raft_consensus.cc:729] T d2d209f53f204d5288490f1c35807422 P 8cbcb48cd2ab4a93956f5392df6c5983 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cbcb48cd2ab4a93956f5392df6c5983, State: Initialized, Role: FOLLOWER
I20260927 12:48:49.289645 15171 consensus_queue.cc:260] T d2d209f53f204d5288490f1c35807422 P 8cbcb48cd2ab4a93956f5392df6c5983 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c9851a903c4436b895d7da878979c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37579 } } peers { permanent_uuid: "8cbcb48cd2ab4a93956f5392df6c5983" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39543 } } peers { permanent_uuid: "104cf2fdae024d4ebe3c93c3ea1afe37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37565 } }
I20260927 12:48:49.290354 15026 heartbeater.cc:503] Master 127.0.98.62:37309 was elected leader, sending a full tablet report...
I20260927 12:48:49.290899 15171 ts_tablet_manager.cc:1428] T d2d209f53f204d5288490f1c35807422 P 8cbcb48cd2ab4a93956f5392df6c5983: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 12:48:49.291971 15172 raft_consensus.cc:348] T d2d209f53f204d5288490f1c35807422 P 87c9851a903c4436b895d7da878979c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c9851a903c4436b895d7da878979c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37579 } } peers { permanent_uuid: "8cbcb48cd2ab4a93956f5392df6c5983" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39543 } } peers { permanent_uuid: "104cf2fdae024d4ebe3c93c3ea1afe37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37565 } }
I20260927 12:48:49.292249 15172 raft_consensus.cc:374] T d2d209f53f204d5288490f1c35807422 P 87c9851a903c4436b895d7da878979c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:48:49.292313 15172 raft_consensus.cc:729] T d2d209f53f204d5288490f1c35807422 P 87c9851a903c4436b895d7da878979c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87c9851a903c4436b895d7da878979c7, State: Initialized, Role: FOLLOWER
W20260927 12:48:49.292344 15027 tablet.cc:2369] T d2d209f53f204d5288490f1c35807422 P 8cbcb48cd2ab4a93956f5392df6c5983: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:48:49.292824 15172 consensus_queue.cc:260] T d2d209f53f204d5288490f1c35807422 P 87c9851a903c4436b895d7da878979c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c9851a903c4436b895d7da878979c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37579 } } peers { permanent_uuid: "8cbcb48cd2ab4a93956f5392df6c5983" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39543 } } peers { permanent_uuid: "104cf2fdae024d4ebe3c93c3ea1afe37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37565 } }
I20260927 12:48:49.293529 15158 heartbeater.cc:503] Master 127.0.98.62:37309 was elected leader, sending a full tablet report...
I20260927 12:48:49.293799 15172 ts_tablet_manager.cc:1428] T d2d209f53f204d5288490f1c35807422 P 87c9851a903c4436b895d7da878979c7: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260927 12:48:49.342962 14889 tablet.cc:2369] T d2d209f53f204d5288490f1c35807422 P 104cf2fdae024d4ebe3c93c3ea1afe37: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 12:48:49.469856 15159 tablet.cc:2369] T d2d209f53f204d5288490f1c35807422 P 87c9851a903c4436b895d7da878979c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:48:49.613488 14894 raft_consensus.cc:482] T d2d209f53f204d5288490f1c35807422 P 104cf2fdae024d4ebe3c93c3ea1afe37 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:48:49.613790 14894 raft_consensus.cc:504] T d2d209f53f204d5288490f1c35807422 P 104cf2fdae024d4ebe3c93c3ea1afe37 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c9851a903c4436b895d7da878979c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37579 } } peers { permanent_uuid: "8cbcb48cd2ab4a93956f5392df6c5983" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39543 } } peers { permanent_uuid: "104cf2fdae024d4ebe3c93c3ea1afe37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37565 } }
I20260927 12:48:49.615088 14894 leader_election.cc:288] T d2d209f53f204d5288490f1c35807422 P 104cf2fdae024d4ebe3c93c3ea1afe37 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87c9851a903c4436b895d7da878979c7 (127.0.98.3:37579), 8cbcb48cd2ab4a93956f5392df6c5983 (127.0.98.2:39543)
I20260927 12:48:49.619493 15177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:49.615294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37579 (local address 127.0.98.1:44123)
0927 12:48:49.615602 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:49.615616 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:48:49.615753 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:48:49.616209 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:48:49.616211 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:48:49.616225 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:48:49.616297 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:49.616303 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:49.617265 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:49.617267 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:49.618375 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:49.618383 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:49.619007 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:49.619010 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:49.619092 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:49.619140 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 12:48:49.619256 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 12:48:49.619329 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":69,"threads_started":1}
I20260927 12:48:49.619498 15167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:49.615791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44123 (local address 127.0.98.3:37579)
0927 12:48:49.615904 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 12:48:49.615908 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:48:49.615920 (+ 12us) server_negotiation.cc:407] Connection header received
0927 12:48:49.615964 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:48:49.615966 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:48:49.616010 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:48:49.616084 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:48:49.616452 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:49.617156 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:49.618478 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:49.618896 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:49.618990 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:49.619052 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 12:48:49.619291 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 12:48:49.619379 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 12:48:49.619509 15169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:49.615510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36141 (local address 127.0.98.2:39543)
0927 12:48:49.615615 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 12:48:49.615620 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:48:49.615725 (+ 105us) server_negotiation.cc:407] Connection header received
0927 12:48:49.615959 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:48:49.615962 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:48:49.616011 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:48:49.616089 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:48:49.616358 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:49.616927 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:49.618382 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:49.619006 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:49.619106 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:49.619166 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 12:48:49.619327 (+ 161us) server_negotiation.cc:299] Negotiation successful
0927 12:48:49.619407 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 12:48:49.619902 15178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:49.615337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39543 (local address 127.0.98.1:36141)
0927 12:48:49.615645 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:49.615658 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:48:49.615794 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:48:49.616107 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:48:49.616110 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:48:49.616128 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:48:49.616251 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:49.616257 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:49.617028 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:49.617031 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:49.618267 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:49.618277 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:49.619115 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:49.619118 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:49.619201 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:49.619222 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 12:48:49.619710 (+ 488us) client_negotiation.cc:240] Negotiation successful
0927 12:48:49.619782 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":1,"thread_start_us":90,"threads_started":1}
I20260927 12:48:49.620466 14980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d209f53f204d5288490f1c35807422" candidate_uuid: "104cf2fdae024d4ebe3c93c3ea1afe37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cbcb48cd2ab4a93956f5392df6c5983" is_pre_election: true
I20260927 12:48:49.620463 15112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d209f53f204d5288490f1c35807422" candidate_uuid: "104cf2fdae024d4ebe3c93c3ea1afe37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87c9851a903c4436b895d7da878979c7" is_pre_election: true
I20260927 12:48:49.620724 14980 raft_consensus.cc:2454] T d2d209f53f204d5288490f1c35807422 P 8cbcb48cd2ab4a93956f5392df6c5983 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 104cf2fdae024d4ebe3c93c3ea1afe37 in term 0.
I20260927 12:48:49.620730 15112 raft_consensus.cc:2454] T d2d209f53f204d5288490f1c35807422 P 87c9851a903c4436b895d7da878979c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 104cf2fdae024d4ebe3c93c3ea1afe37 in term 0.
I20260927 12:48:49.621121 14776 leader_election.cc:302] T d2d209f53f204d5288490f1c35807422 P 104cf2fdae024d4ebe3c93c3ea1afe37 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 104cf2fdae024d4ebe3c93c3ea1afe37, 8cbcb48cd2ab4a93956f5392df6c5983; no voters:
I20260927 12:48:49.621397 14894 raft_consensus.cc:2781] T d2d209f53f204d5288490f1c35807422 P 104cf2fdae024d4ebe3c93c3ea1afe37 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:48:49.621479 14894 raft_consensus.cc:482] T d2d209f53f204d5288490f1c35807422 P 104cf2fdae024d4ebe3c93c3ea1afe37 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:48:49.621510 14894 raft_consensus.cc:3037] T d2d209f53f204d5288490f1c35807422 P 104cf2fdae024d4ebe3c93c3ea1afe37 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:48:49.622555 14894 raft_consensus.cc:504] T d2d209f53f204d5288490f1c35807422 P 104cf2fdae024d4ebe3c93c3ea1afe37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c9851a903c4436b895d7da878979c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37579 } } peers { permanent_uuid: "8cbcb48cd2ab4a93956f5392df6c5983" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39543 } } peers { permanent_uuid: "104cf2fdae024d4ebe3c93c3ea1afe37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37565 } }
I20260927 12:48:49.622897 14894 leader_election.cc:288] T d2d209f53f204d5288490f1c35807422 P 104cf2fdae024d4ebe3c93c3ea1afe37 [CANDIDATE]: Term 1 election: Requested vote from peers 87c9851a903c4436b895d7da878979c7 (127.0.98.3:37579), 8cbcb48cd2ab4a93956f5392df6c5983 (127.0.98.2:39543)
I20260927 12:48:49.623252 14980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d209f53f204d5288490f1c35807422" candidate_uuid: "104cf2fdae024d4ebe3c93c3ea1afe37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cbcb48cd2ab4a93956f5392df6c5983"
I20260927 12:48:49.623252 15112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d209f53f204d5288490f1c35807422" candidate_uuid: "104cf2fdae024d4ebe3c93c3ea1afe37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87c9851a903c4436b895d7da878979c7"
I20260927 12:48:49.623385 15112 raft_consensus.cc:3037] T d2d209f53f204d5288490f1c35807422 P 87c9851a903c4436b895d7da878979c7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:48:49.623385 14980 raft_consensus.cc:3037] T d2d209f53f204d5288490f1c35807422 P 8cbcb48cd2ab4a93956f5392df6c5983 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:48:49.624588 14980 raft_consensus.cc:2454] T d2d209f53f204d5288490f1c35807422 P 8cbcb48cd2ab4a93956f5392df6c5983 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 104cf2fdae024d4ebe3c93c3ea1afe37 in term 1.
I20260927 12:48:49.624588 15112 raft_consensus.cc:2454] T d2d209f53f204d5288490f1c35807422 P 87c9851a903c4436b895d7da878979c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 104cf2fdae024d4ebe3c93c3ea1afe37 in term 1.
I20260927 12:48:49.624850 14775 leader_election.cc:302] T d2d209f53f204d5288490f1c35807422 P 104cf2fdae024d4ebe3c93c3ea1afe37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 104cf2fdae024d4ebe3c93c3ea1afe37, 87c9851a903c4436b895d7da878979c7; no voters:
I20260927 12:48:49.625056 14894 raft_consensus.cc:2781] T d2d209f53f204d5288490f1c35807422 P 104cf2fdae024d4ebe3c93c3ea1afe37 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:48:49.625212 14894 raft_consensus.cc:686] T d2d209f53f204d5288490f1c35807422 P 104cf2fdae024d4ebe3c93c3ea1afe37 [term 1 LEADER]: Becoming Leader. State: Replica: 104cf2fdae024d4ebe3c93c3ea1afe37, State: Running, Role: LEADER
I20260927 12:48:49.625442 14894 consensus_queue.cc:237] T d2d209f53f204d5288490f1c35807422 P 104cf2fdae024d4ebe3c93c3ea1afe37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c9851a903c4436b895d7da878979c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37579 } } peers { permanent_uuid: "8cbcb48cd2ab4a93956f5392df6c5983" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39543 } } peers { permanent_uuid: "104cf2fdae024d4ebe3c93c3ea1afe37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37565 } }
I20260927 12:48:49.628100 14699 catalog_manager.cc:5495] T d2d209f53f204d5288490f1c35807422 P 104cf2fdae024d4ebe3c93c3ea1afe37 reported cstate change: term changed from 0 to 1, leader changed from <none> to 104cf2fdae024d4ebe3c93c3ea1afe37 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "104cf2fdae024d4ebe3c93c3ea1afe37" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c9851a903c4436b895d7da878979c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8cbcb48cd2ab4a93956f5392df6c5983" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "104cf2fdae024d4ebe3c93c3ea1afe37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37565 } health_report { overall_health: HEALTHY } } }
I20260927 12:48:49.637859 15182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:49.634762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37676 (local address 127.0.98.1:37565)
0927 12:48:49.634999 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 12:48:49.635006 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 12:48:49.635017 (+ 11us) server_negotiation.cc:407] Connection header received
0927 12:48:49.635076 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:48:49.635078 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:48:49.635126 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:48:49.635217 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:48:49.635383 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:49.635900 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:49.636668 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:49.636871 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:49.636934 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:49.637162 (+ 228us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:48:49.637570 (+ 408us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:48:49.637648 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 12:48:49.637687 (+ 39us) server_negotiation.cc:299] Negotiation successful
0927 12:48:49.637732 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":96,"threads_started":1}
I20260927 12:48:49.641408 15112 raft_consensus.cc:1261] T d2d209f53f204d5288490f1c35807422 P 87c9851a903c4436b895d7da878979c7 [term 1 FOLLOWER]: Refusing update from remote peer 104cf2fdae024d4ebe3c93c3ea1afe37: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:48:49.641409 14980 raft_consensus.cc:1261] T d2d209f53f204d5288490f1c35807422 P 8cbcb48cd2ab4a93956f5392df6c5983 [term 1 FOLLOWER]: Refusing update from remote peer 104cf2fdae024d4ebe3c93c3ea1afe37: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:48:49.642031 14894 consensus_queue.cc:1035] T d2d209f53f204d5288490f1c35807422 P 104cf2fdae024d4ebe3c93c3ea1afe37 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87c9851a903c4436b895d7da878979c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:48:49.642230 15179 consensus_queue.cc:1035] T d2d209f53f204d5288490f1c35807422 P 104cf2fdae024d4ebe3c93c3ea1afe37 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cbcb48cd2ab4a93956f5392df6c5983" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:48:49.649188 15183 mvcc.cc:205] Tried to move back new op lower bound from 7333942598205403136 to 7333942598148034560. Current Snapshot: MvccSnapshot[applied={T|T < 7333942598205403136}]
I20260927 12:48:49.650079 15185 mvcc.cc:205] Tried to move back new op lower bound from 7333942598205403136 to 7333942598148034560. Current Snapshot: MvccSnapshot[applied={T|T < 7333942598205403136}]
I20260927 12:48:49.654250 15184 mvcc.cc:205] Tried to move back new op lower bound from 7333942598205403136 to 7333942598148034560. Current Snapshot: MvccSnapshot[applied={T|T < 7333942598205403136 or (T in {7333942598205403136})}]
I20260927 12:48:49.656669 14700 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:45266:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:48:49.656817 14700 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:45266:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:48:49.659016 14700 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 76a78d2ebcd0446ebdafa41ff473c402: Sending DeleteTablet for 3 replicas of tablet d2d209f53f204d5288490f1c35807422
I20260927 12:48:49.659679 15092 tablet_service.cc:1514] Processing DeleteTablet for tablet d2d209f53f204d5288490f1c35807422 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:48:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37030
I20260927 12:48:49.659718 14960 tablet_service.cc:1514] Processing DeleteTablet for tablet d2d209f53f204d5288490f1c35807422 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:48:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40944
I20260927 12:48:49.659885 14822 tablet_service.cc:1514] Processing DeleteTablet for tablet d2d209f53f204d5288490f1c35807422 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:48:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37672
I20260927 12:48:49.660172 15195 tablet_replica.cc:330] T d2d209f53f204d5288490f1c35807422 P 104cf2fdae024d4ebe3c93c3ea1afe37: stopping tablet replica
I20260927 12:48:49.660311 15195 raft_consensus.cc:2229] T d2d209f53f204d5288490f1c35807422 P 104cf2fdae024d4ebe3c93c3ea1afe37 [term 1 LEADER]: Raft consensus shutting down.
I20260927 12:48:49.660580 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 14738
I20260927 12:48:49.660632 15195 raft_consensus.cc:2258] T d2d209f53f204d5288490f1c35807422 P 104cf2fdae024d4ebe3c93c3ea1afe37 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:48:49.661863 15194 tablet_replica.cc:330] T d2d209f53f204d5288490f1c35807422 P 87c9851a903c4436b895d7da878979c7: stopping tablet replica
I20260927 12:48:49.662199 15194 raft_consensus.cc:2229] T d2d209f53f204d5288490f1c35807422 P 87c9851a903c4436b895d7da878979c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:48:49.662506 15194 raft_consensus.cc:2258] T d2d209f53f204d5288490f1c35807422 P 87c9851a903c4436b895d7da878979c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:48:49.662865 15193 tablet_replica.cc:330] T d2d209f53f204d5288490f1c35807422 P 8cbcb48cd2ab4a93956f5392df6c5983: stopping tablet replica
I20260927 12:48:49.663148 15193 raft_consensus.cc:2229] T d2d209f53f204d5288490f1c35807422 P 8cbcb48cd2ab4a93956f5392df6c5983 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:48:49.663436 15193 raft_consensus.cc:2258] T d2d209f53f204d5288490f1c35807422 P 8cbcb48cd2ab4a93956f5392df6c5983 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:48:49.664161 15194 ts_tablet_manager.cc:1905] T d2d209f53f204d5288490f1c35807422 P 87c9851a903c4436b895d7da878979c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:48:49.664572 15193 ts_tablet_manager.cc:1905] T d2d209f53f204d5288490f1c35807422 P 8cbcb48cd2ab4a93956f5392df6c5983: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:48:49.666694 15193 ts_tablet_manager.cc:1918] T d2d209f53f204d5288490f1c35807422 P 8cbcb48cd2ab4a93956f5392df6c5983: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:48:49.666790 15193 log.cc:1196] T d2d209f53f204d5288490f1c35807422 P 8cbcb48cd2ab4a93956f5392df6c5983: Deleting WAL directory at /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-1/wal/wals/d2d209f53f204d5288490f1c35807422
I20260927 12:48:49.667060 15193 ts_tablet_manager.cc:1939] T d2d209f53f204d5288490f1c35807422 P 8cbcb48cd2ab4a93956f5392df6c5983: Deleting consensus metadata
I20260927 12:48:49.667515 15194 ts_tablet_manager.cc:1918] T d2d209f53f204d5288490f1c35807422 P 87c9851a903c4436b895d7da878979c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:48:49.667624 15194 log.cc:1196] T d2d209f53f204d5288490f1c35807422 P 87c9851a903c4436b895d7da878979c7: Deleting WAL directory at /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-2/wal/wals/d2d209f53f204d5288490f1c35807422
I20260927 12:48:49.667937 15194 ts_tablet_manager.cc:1939] T d2d209f53f204d5288490f1c35807422 P 87c9851a903c4436b895d7da878979c7: Deleting consensus metadata
I20260927 12:48:49.668335 14686 catalog_manager.cc:4841] TS 8cbcb48cd2ab4a93956f5392df6c5983 (127.0.98.2:39543): tablet d2d209f53f204d5288490f1c35807422 (table test-table [id=30d7b1b4b6074e5f80a6fce87c95f341]) successfully deleted
W20260927 12:48:49.668452 14684 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37565 (error 108)
I20260927 12:48:49.668705 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 14893
I20260927 12:48:49.669269 14685 catalog_manager.cc:4841] TS 87c9851a903c4436b895d7da878979c7 (127.0.98.3:37579): tablet d2d209f53f204d5288490f1c35807422 (table test-table [id=30d7b1b4b6074e5f80a6fce87c95f341]) successfully deleted
I20260927 12:48:49.674108 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 15029
I20260927 12:48:49.675093 15168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:49.674783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37565 (local address 127.0.0.1:37686)
0927 12:48:49.674933 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:49.675005 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260927 12:48:49.675547 14684 catalog_manager.cc:4584] TS 104cf2fdae024d4ebe3c93c3ea1afe37 (127.0.98.1:37565): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d2d209f53f204d5288490f1c35807422: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37565: connect: Connection refused (error 111)
I20260927 12:48:49.679346 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 14667
I20260927 12:48:49.684582 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37309
--webserver_interface=127.0.98.62
--webserver_port=44923
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37309
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:48:49.793051 15197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:48:49.793308 15197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:48:49.793347 15197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:48:49.797361 15197 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:48:49.797425 15197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:48:49.797443 15197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:48:49.797458 15197 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:48:49.797472 15197 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:48:49.802798 15197 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37309
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=44923
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:48:49.803894 15197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:48:49.804953 15197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:48:49.813782 15202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:48:49.813736 15205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:48:49.813733 15203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:48:49.813951 15197 server_base.cc:1034] running on GCE node
I20260927 12:48:49.814733 15197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:48:49.815831 15197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:48:49.816989 15197 hybrid_clock.cc:648] HybridClock initialized: now 1790513329816962 us; error 37 us; skew 500 ppm
Sep 27 12:48:49 dist-test-slave-lkn4 krb5kdc[14119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513329, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:48:49.819676 15197 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:48:49.820508 15197 webserver.cc:460] Webserver started at http://127.0.98.62:44923/ using document root <none> and password file <none>
I20260927 12:48:49.821058 15197 fs_manager.cc:359] Metadata directory not provided
I20260927 12:48:49.821122 15197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:48:49.824775 15197 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:48:49.826853 15212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:48:49.828019 15197 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:48:49.828126 15197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/master-0/wal
uuid: "76a78d2ebcd0446ebdafa41ff473c402"
format_stamp: "Formatted at 2026-09-27 12:48:47 on dist-test-slave-lkn4"
server_key: "f7aafd8aa5d0a262908a480aa93c0a1f"
server_key_iv: "b340ad9cad868032c70ae03cb94d9054"
server_key_version: "encryptionkey@0"
I20260927 12:48:49.828644 15197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:48:49.858210 15197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:48:49.861287 15197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:48:49.861466 15197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:48:49.868041 15197 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37309
I20260927 12:48:49.868279 15264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37309 every 8 connection(s)
I20260927 12:48:49.869093 15197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:48:49.870527 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 15197
I20260927 12:48:49.870950 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:37565
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:37309
--webserver_port=43289
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 12:48:49.873771 15265 sys_catalog.cc:263] Verifying existing consensus state
I20260927 12:48:49.875420 15265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76a78d2ebcd0446ebdafa41ff473c402: Bootstrap starting.
I20260927 12:48:49.891747 15265 log.cc:824] T 00000000000000000000000000000000 P 76a78d2ebcd0446ebdafa41ff473c402: Log is configured to *not* fsync() on all Append() calls
I20260927 12:48:49.902848 15265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76a78d2ebcd0446ebdafa41ff473c402: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 12:48:49.903245 15265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76a78d2ebcd0446ebdafa41ff473c402: Bootstrap complete.
I20260927 12:48:49.907178 15265 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76a78d2ebcd0446ebdafa41ff473c402 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76a78d2ebcd0446ebdafa41ff473c402" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37309 } }
I20260927 12:48:49.907603 15265 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76a78d2ebcd0446ebdafa41ff473c402 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76a78d2ebcd0446ebdafa41ff473c402, State: Initialized, Role: FOLLOWER
I20260927 12:48:49.908129 15265 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76a78d2ebcd0446ebdafa41ff473c402 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76a78d2ebcd0446ebdafa41ff473c402" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37309 } }
I20260927 12:48:49.908250 15265 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76a78d2ebcd0446ebdafa41ff473c402 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:48:49.908313 15265 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76a78d2ebcd0446ebdafa41ff473c402 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:48:49.908403 15265 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76a78d2ebcd0446ebdafa41ff473c402 [term 2 FOLLOWER]: Advancing to term 3
I20260927 12:48:49.910320 15265 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76a78d2ebcd0446ebdafa41ff473c402 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76a78d2ebcd0446ebdafa41ff473c402" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37309 } }
I20260927 12:48:49.910629 15265 leader_election.cc:302] T 00000000000000000000000000000000 P 76a78d2ebcd0446ebdafa41ff473c402 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76a78d2ebcd0446ebdafa41ff473c402; no voters:
I20260927 12:48:49.910928 15265 leader_election.cc:288] T 00000000000000000000000000000000 P 76a78d2ebcd0446ebdafa41ff473c402 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 12:48:49.911026 15269 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76a78d2ebcd0446ebdafa41ff473c402 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 12:48:49.911211 15269 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76a78d2ebcd0446ebdafa41ff473c402 [term 3 LEADER]: Becoming Leader. State: Replica: 76a78d2ebcd0446ebdafa41ff473c402, State: Running, Role: LEADER
I20260927 12:48:49.911509 15269 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76a78d2ebcd0446ebdafa41ff473c402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76a78d2ebcd0446ebdafa41ff473c402" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37309 } }
I20260927 12:48:49.912149 15265 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76a78d2ebcd0446ebdafa41ff473c402 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:48:49.912467 15269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76a78d2ebcd0446ebdafa41ff473c402 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76a78d2ebcd0446ebdafa41ff473c402. Latest consensus state: current_term: 3 leader_uuid: "76a78d2ebcd0446ebdafa41ff473c402" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76a78d2ebcd0446ebdafa41ff473c402" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37309 } } }
I20260927 12:48:49.912689 15269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76a78d2ebcd0446ebdafa41ff473c402 [sys.catalog]: This master's current role is: LEADER
I20260927 12:48:49.913770 15273 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:48:49.913872 15270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76a78d2ebcd0446ebdafa41ff473c402 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "76a78d2ebcd0446ebdafa41ff473c402" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76a78d2ebcd0446ebdafa41ff473c402" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37309 } } }
I20260927 12:48:49.914033 15270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76a78d2ebcd0446ebdafa41ff473c402 [sys.catalog]: This master's current role is: LEADER
I20260927 12:48:49.918718 15273 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:48:49.919381 15273 catalog_manager.cc:1259] Loaded cluster ID: 6c455548fd1a40088a72a562cfb393ba
I20260927 12:48:49.919447 15273 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260927 12:48:49.920457 15273 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f2128cf39ad google::LogMessage::Fail() at ??:0
@ 0x7f2128cf688a google::LogMessage::SendToLog() at ??:0
@ 0x7f2128cf34de google::LogMessage::Flush() at ??:0
@ 0x7f2128cf4a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f2133632fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f2133633d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f2133630579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f2133674640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f212af453d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f212af45c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f212af473f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f212af36512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f212c57a6db start_thread at ??:0
@ 0x7f21277f161f clone at ??:0
W20260927 12:48:50.006912 15267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:48:50.007313 15267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:48:50.007390 15267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:48:50.013489 15267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:48:50.013578 15267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:48:50.013667 15267 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:48:50.021814 15267 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:37565
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=43289
--tserver_master_addrs=127.0.98.62:37309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15267
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:48:50.023368 15267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:48:50.024647 15267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:48:50.035751 15306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:48:50.035821 15307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:48:50.035751 15309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:48:50.036273 15267 server_base.cc:1034] running on GCE node
I20260927 12:48:50.036698 15267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:48:50.037335 15267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:48:50.038491 15267 hybrid_clock.cc:648] HybridClock initialized: now 1790513330038454 us; error 55 us; skew 500 ppm
Sep 27 12:48:50 dist-test-slave-lkn4 krb5kdc[14119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513330, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:48:50.041132 15267 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 12:48:50.041975 15267 webserver.cc:460] Webserver started at http://127.0.98.1:43289/ using document root <none> and password file <none>
I20260927 12:48:50.042511 15267 fs_manager.cc:359] Metadata directory not provided
I20260927 12:48:50.042568 15267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:48:50.046288 15267 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260927 12:48:50.048441 15316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:48:50.049777 15267 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:48:50.049896 15267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-0/wal
uuid: "104cf2fdae024d4ebe3c93c3ea1afe37"
format_stamp: "Formatted at 2026-09-27 12:48:47 on dist-test-slave-lkn4"
server_key: "7999efb846f0c33cee2dd92b36225e5d"
server_key_iv: "25a33bbfb862beab983e0ff1d9a486a5"
server_key_version: "encryptionkey@0"
I20260927 12:48:50.050453 15267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:48:50.069839 15267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:48:50.073027 15267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:48:50.073203 15267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:48:50.073766 15267 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:48:50.074922 15323 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 12:48:50.080624 15267 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 12:48:50.080694 15267 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260927 12:48:50.080760 15267 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 12:48:50.082499 15267 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 12:48:50.082549 15267 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260927 12:48:50.082921 15323 tablet_bootstrap.cc:492] T d2d209f53f204d5288490f1c35807422 P 104cf2fdae024d4ebe3c93c3ea1afe37: Bootstrap starting.
I20260927 12:48:50.092177 15267 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37565
I20260927 12:48:50.092578 15430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37565 every 8 connection(s)
I20260927 12:48:50.093235 15267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:48:50.097890 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 15267
I20260927 12:48:50.098309 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:39543
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:37309
--webserver_port=46479
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 12:48:50.105043 15323 log.cc:824] T d2d209f53f204d5288490f1c35807422 P 104cf2fdae024d4ebe3c93c3ea1afe37: Log is configured to *not* fsync() on all Append() calls
I20260927 12:48:50.113848 15323 tablet_bootstrap.cc:492] T d2d209f53f204d5288490f1c35807422 P 104cf2fdae024d4ebe3c93c3ea1afe37: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 12:48:50.114251 15323 tablet_bootstrap.cc:492] T d2d209f53f204d5288490f1c35807422 P 104cf2fdae024d4ebe3c93c3ea1afe37: Bootstrap complete.
I20260927 12:48:50.115537 15323 ts_tablet_manager.cc:1397] T d2d209f53f204d5288490f1c35807422 P 104cf2fdae024d4ebe3c93c3ea1afe37: Time spent bootstrapping tablet: real 0.033s user 0.022s sys 0.009s
I20260927 12:48:50.118546 15323 raft_consensus.cc:348] T d2d209f53f204d5288490f1c35807422 P 104cf2fdae024d4ebe3c93c3ea1afe37 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c9851a903c4436b895d7da878979c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37579 } } peers { permanent_uuid: "8cbcb48cd2ab4a93956f5392df6c5983" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39543 } } peers { permanent_uuid: "104cf2fdae024d4ebe3c93c3ea1afe37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37565 } }
I20260927 12:48:50.119087 15323 raft_consensus.cc:729] T d2d209f53f204d5288490f1c35807422 P 104cf2fdae024d4ebe3c93c3ea1afe37 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 104cf2fdae024d4ebe3c93c3ea1afe37, State: Initialized, Role: FOLLOWER
I20260927 12:48:50.119593 15323 consensus_queue.cc:260] T d2d209f53f204d5288490f1c35807422 P 104cf2fdae024d4ebe3c93c3ea1afe37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c9851a903c4436b895d7da878979c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37579 } } peers { permanent_uuid: "8cbcb48cd2ab4a93956f5392df6c5983" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39543 } } peers { permanent_uuid: "104cf2fdae024d4ebe3c93c3ea1afe37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37565 } }
I20260927 12:48:50.120623 15323 ts_tablet_manager.cc:1428] T d2d209f53f204d5288490f1c35807422 P 104cf2fdae024d4ebe3c93c3ea1afe37: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
W20260927 12:48:50.172989 15433 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:48:50.095082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37309 (local address 127.0.98.1:59371)
0927 12:48:50.095542 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:50.095594 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 12:48:50.096371 (+ 777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:48:50.171944 (+ 75573us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37309: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":104,"threads_started":1}
W20260927 12:48:50.173579 15431 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37309 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:37309: Client connection negotiation failed: client connection to 127.0.98.62:37309: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 12:48:50.174427 15433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:50.174148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37309 (local address 127.0.98.1:35663)
0927 12:48:50.174300 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:50.174369 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 12:48:50.175266 15433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:50.174988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37309 (local address 127.0.98.1:46577)
0927 12:48:50.175148 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:50.175201 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260927 12:48:50.175783 15431 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 12:48:50.209812 15434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:48:50.210108 15434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:48:50.210181 15434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:48:50.214238 15434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:48:50.214305 15434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:48:50.214386 15434 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:48:50.219669 15434 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:39543
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=46479
--tserver_master_addrs=127.0.98.62:37309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15434
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:48:50.220728 15434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:48:50.221738 15434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:48:50.231248 15442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:48:50.231244 15441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:48:50.231244 15444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:48:50.231698 15434 server_base.cc:1034] running on GCE node
I20260927 12:48:50.232097 15434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:48:50.232723 15434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:48:50.233877 15434 hybrid_clock.cc:648] HybridClock initialized: now 1790513330233859 us; error 31 us; skew 500 ppm
Sep 27 12:48:50 dist-test-slave-lkn4 krb5kdc[14119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513330, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:48:50.236603 15434 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 12:48:50.237468 15434 webserver.cc:460] Webserver started at http://127.0.98.2:46479/ using document root <none> and password file <none>
I20260927 12:48:50.238009 15434 fs_manager.cc:359] Metadata directory not provided
I20260927 12:48:50.238099 15434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:48:50.241814 15434 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:48:50.243899 15451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:48:50.245042 15434 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 12:48:50.245151 15434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-1/wal
uuid: "8cbcb48cd2ab4a93956f5392df6c5983"
format_stamp: "Formatted at 2026-09-27 12:48:47 on dist-test-slave-lkn4"
server_key: "6050048a83049ae6e551f24e3f21eab9"
server_key_iv: "d51ae3f054663ff1319b0f5dec057945"
server_key_version: "encryptionkey@0"
I20260927 12:48:50.245658 15434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:48:50.255664 15434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:48:50.258447 15434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:48:50.258616 15434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:48:50.259163 15434 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:48:50.259975 15434 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:48:50.260020 15434 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:48:50.260070 15434 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:48:50.260098 15434 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:48:50.268404 15434 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39543
I20260927 12:48:50.268447 15564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39543 every 8 connection(s)
I20260927 12:48:50.269443 15434 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:48:50.272881 15567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:50.271481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37309 (local address 127.0.98.2:51443)
0927 12:48:50.272088 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:50.272189 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":102,"threads_started":1}
W20260927 12:48:50.273249 15565 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37309 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:37309: Client connection negotiation failed: client connection to 127.0.98.62:37309: connect: Connection refused (error 111)
I20260927 12:48:50.273772 15567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:50.273592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37309 (local address 127.0.98.2:44305)
0927 12:48:50.273681 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:50.273722 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 12:48:50.274269 15567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:50.274106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37309 (local address 127.0.98.2:36809)
0927 12:48:50.274186 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:50.274224 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 12:48:50.274415 15565 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 12:48:50.275477 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 15434
I20260927 12:48:50.275836 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:37579
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:37309
--webserver_port=35697
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:48:50.383710 15568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:48:50.383976 15568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:48:50.384018 15568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:48:50.388038 15568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:48:50.388108 15568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:48:50.388175 15568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:48:50.393443 15568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:37579
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=35697
--tserver_master_addrs=127.0.98.62:37309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:48:50.394552 15568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:48:50.395560 15568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:48:50.404700 15574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:48:50.404700 15576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:48:50.404706 15573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:48:50.404985 15568 server_base.cc:1034] running on GCE node
I20260927 12:48:50.405375 15568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:48:50.406006 15568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:48:50.407181 15568 hybrid_clock.cc:648] HybridClock initialized: now 1790513330407162 us; error 30 us; skew 500 ppm
Sep 27 12:48:50 dist-test-slave-lkn4 krb5kdc[14119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513330, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:48:50.409721 15568 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 12:48:50.410635 15568 webserver.cc:460] Webserver started at http://127.0.98.3:35697/ using document root <none> and password file <none>
I20260927 12:48:50.411190 15568 fs_manager.cc:359] Metadata directory not provided
I20260927 12:48:50.411247 15568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:48:50.414960 15568 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.001s
I20260927 12:48:50.417057 15583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:48:50.418455 15568 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 12:48:50.418558 15568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-2/wal
uuid: "87c9851a903c4436b895d7da878979c7"
format_stamp: "Formatted at 2026-09-27 12:48:48 on dist-test-slave-lkn4"
server_key: "c8ecfc2f6ec145221221b98795cc6cee"
server_key_iv: "424022327d0b2b00281f1069faf133aa"
server_key_version: "encryptionkey@0"
I20260927 12:48:50.419117 15568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:48:50.433390 15568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:48:50.436467 15568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:48:50.436642 15568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:48:50.437295 15568 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:48:50.438271 15568 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:48:50.438328 15568 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:48:50.438371 15568 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:48:50.438400 15568 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:48:50.446926 15568 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37579
I20260927 12:48:50.446964 15696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37579 every 8 connection(s)
I20260927 12:48:50.447949 15568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:48:50.451646 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:50.450183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37309 (local address 127.0.98.3:54505)
0927 12:48:50.450826 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:50.450939 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":112,"threads_started":1}
I20260927 12:48:50.451754 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 15568
W20260927 12:48:50.452195 15697 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37309 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:37309: Client connection negotiation failed: client connection to 127.0.98.62:37309: connect: Connection refused (error 111)
I20260927 12:48:50.452804 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:50.452568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37309 (local address 127.0.98.3:57887)
0927 12:48:50.452683 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:50.452735 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 12:48:50.453354 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:50.453158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37309 (local address 127.0.98.3:36237)
0927 12:48:50.453252 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:50.453294 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 12:48:50.453490 15697 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 12:48:51.176779 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:51.176358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37309 (local address 127.0.98.1:55897)
0927 12:48:51.176623 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:51.176679 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":75,"threads_started":1}
I20260927 12:48:51.275362 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:51.274935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37309 (local address 127.0.98.2:51495)
0927 12:48:51.275176 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:51.275232 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":81,"threads_started":1}
I20260927 12:48:51.454489 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:51.454029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37309 (local address 127.0.98.3:60817)
0927 12:48:51.454325 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:51.454388 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":78,"threads_started":1}
I20260927 12:48:51.528551 15704 raft_consensus.cc:482] T d2d209f53f204d5288490f1c35807422 P 104cf2fdae024d4ebe3c93c3ea1afe37 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:48:51.528731 15704 raft_consensus.cc:504] T d2d209f53f204d5288490f1c35807422 P 104cf2fdae024d4ebe3c93c3ea1afe37 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c9851a903c4436b895d7da878979c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37579 } } peers { permanent_uuid: "8cbcb48cd2ab4a93956f5392df6c5983" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39543 } } peers { permanent_uuid: "104cf2fdae024d4ebe3c93c3ea1afe37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37565 } }
I20260927 12:48:51.529791 15704 leader_election.cc:288] T d2d209f53f204d5288490f1c35807422 P 104cf2fdae024d4ebe3c93c3ea1afe37 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 87c9851a903c4436b895d7da878979c7 (127.0.98.3:37579), 8cbcb48cd2ab4a93956f5392df6c5983 (127.0.98.2:39543)
Sep 27 12:48:51 dist-test-slave-lkn4 krb5kdc[14119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513330, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 27 12:48:51 dist-test-slave-lkn4 krb5kdc[14119](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513330, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260927 12:48:51.543283 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:51.529982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37579 (local address 127.0.98.1:59083)
0927 12:48:51.530181 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:51.530197 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:48:51.530316 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:48:51.531987 (+ 1671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:48:51.531993 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:48:51.532467 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:48:51.532978 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:51.532993 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:51.534430 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:51.534434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:51.535366 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:51.535375 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:51.536051 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:51.536054 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:51.536296 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:51.536906 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:48:51.536931 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:48:51.538842 (+ 1911us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:48:51.540870 (+ 2028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:48:51.540873 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:48:51.540875 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:48:51.541150 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:48:51.541487 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:48:51.541490 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:48:51.541492 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:48:51.541633 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:48:51.542304 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:48:51.542310 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:48:51.542570 (+ 260us) client_negotiation.cc:769] Sending connection context
0927 12:48:51.542736 (+ 166us) client_negotiation.cc:240] Negotiation successful
0927 12:48:51.543094 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":168,"spinlock_wait_cycles":53760}
I20260927 12:48:51.543303 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:51.530218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41723 (local address 127.0.98.2:39543)
0927 12:48:51.530649 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0927 12:48:51.530669 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 12:48:51.530700 (+ 31us) server_negotiation.cc:407] Connection header received
0927 12:48:51.531321 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:48:51.531326 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:48:51.531636 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:48:51.531897 (+ 261us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:48:51.533142 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:51.534259 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:51.535478 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:51.535936 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:51.536086 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:51.538731 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:48:51.538738 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:48:51.538756 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:48:51.538793 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:48:51.540659 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:48:51.541266 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:48:51.541271 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:48:51.541275 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:48:51.541400 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:48:51.541731 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:48:51.541734 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:48:51.541735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:48:51.542199 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:48:51.542343 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 12:48:51.542859 (+ 516us) server_negotiation.cc:299] Negotiation successful
0927 12:48:51.543112 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":114,"threads_started":1}
I20260927 12:48:51.543311 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:51.530228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59083 (local address 127.0.98.3:37579)
0927 12:48:51.530574 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0927 12:48:51.530596 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 12:48:51.530634 (+ 38us) server_negotiation.cc:407] Connection header received
0927 12:48:51.531293 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:48:51.531300 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:48:51.531636 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:48:51.531886 (+ 250us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:48:51.533141 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:51.534304 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:51.535480 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:48:51.535934 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:48:51.536087 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:51.539063 (+ 2976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:48:51.539069 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:48:51.539081 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:48:51.539131 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:48:51.540749 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:48:51.541264 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:48:51.541269 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:48:51.541273 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:48:51.541397 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:48:51.541717 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:48:51.541720 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:48:51.541723 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:48:51.542199 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:48:51.542350 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 12:48:51.542859 (+ 509us) server_negotiation.cc:299] Negotiation successful
0927 12:48:51.543111 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":81,"threads_started":1}
I20260927 12:48:51.543323 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:51.530065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39543 (local address 127.0.98.1:41723)
0927 12:48:51.530408 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:51.530421 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:48:51.530523 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:48:51.532241 (+ 1718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:48:51.532243 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:48:51.532485 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:48:51.532977 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:51.532993 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:51.534425 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:51.534428 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:51.535366 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:48:51.535375 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:48:51.536051 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:48:51.536054 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:48:51.536297 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:48:51.536940 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:48:51.536953 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:48:51.538473 (+ 1520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:48:51.540794 (+ 2321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:48:51.540800 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:48:51.540815 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:48:51.541144 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:48:51.541490 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:48:51.541493 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:48:51.541495 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:48:51.541640 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:48:51.542305 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:48:51.542311 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:48:51.542568 (+ 257us) client_negotiation.cc:769] Sending connection context
0927 12:48:51.542735 (+ 167us) client_negotiation.cc:240] Negotiation successful
0927 12:48:51.543170 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"mutex_wait_us":293,"spinlock_wait_cycles":16640,"thread_start_us":69,"threads_started":1}
W20260927 12:48:51.546121 15319 leader_election.cc:341] T d2d209f53f204d5288490f1c35807422 P 104cf2fdae024d4ebe3c93c3ea1afe37 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8cbcb48cd2ab4a93956f5392df6c5983 (127.0.98.2:39543): Not found: Tablet not found: d2d209f53f204d5288490f1c35807422
W20260927 12:48:51.546406 15318 leader_election.cc:341] T d2d209f53f204d5288490f1c35807422 P 104cf2fdae024d4ebe3c93c3ea1afe37 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 87c9851a903c4436b895d7da878979c7 (127.0.98.3:37579): Not found: Tablet not found: d2d209f53f204d5288490f1c35807422
I20260927 12:48:51.546504 15318 leader_election.cc:302] T d2d209f53f204d5288490f1c35807422 P 104cf2fdae024d4ebe3c93c3ea1afe37 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 104cf2fdae024d4ebe3c93c3ea1afe37; no voters: 87c9851a903c4436b895d7da878979c7, 8cbcb48cd2ab4a93956f5392df6c5983
I20260927 12:48:51.546763 15704 raft_consensus.cc:2726] T d2d209f53f204d5288490f1c35807422 P 104cf2fdae024d4ebe3c93c3ea1afe37 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 12:48:52.177951 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:52.177503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37309 (local address 127.0.98.1:40971)
0927 12:48:52.177799 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:52.177857 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":107,"threads_started":1}
I20260927 12:48:52.276407 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:52.276038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37309 (local address 127.0.98.2:40937)
0927 12:48:52.276250 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:52.276319 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":65,"threads_started":1}
I20260927 12:48:52.455590 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:52.455180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37309 (local address 127.0.98.3:33991)
0927 12:48:52.455438 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:52.455495 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":84,"threads_started":1}
I20260927 12:48:53.159829 15711 raft_consensus.cc:482] T d2d209f53f204d5288490f1c35807422 P 104cf2fdae024d4ebe3c93c3ea1afe37 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:48:53.159962 15711 raft_consensus.cc:504] T d2d209f53f204d5288490f1c35807422 P 104cf2fdae024d4ebe3c93c3ea1afe37 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c9851a903c4436b895d7da878979c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37579 } } peers { permanent_uuid: "8cbcb48cd2ab4a93956f5392df6c5983" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39543 } } peers { permanent_uuid: "104cf2fdae024d4ebe3c93c3ea1afe37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37565 } }
I20260927 12:48:53.160377 15711 leader_election.cc:288] T d2d209f53f204d5288490f1c35807422 P 104cf2fdae024d4ebe3c93c3ea1afe37 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 87c9851a903c4436b895d7da878979c7 (127.0.98.3:37579), 8cbcb48cd2ab4a93956f5392df6c5983 (127.0.98.2:39543)
W20260927 12:48:53.161185 15318 leader_election.cc:341] T d2d209f53f204d5288490f1c35807422 P 104cf2fdae024d4ebe3c93c3ea1afe37 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 87c9851a903c4436b895d7da878979c7 (127.0.98.3:37579): Not found: Tablet not found: d2d209f53f204d5288490f1c35807422
W20260927 12:48:53.161298 15319 leader_election.cc:341] T d2d209f53f204d5288490f1c35807422 P 104cf2fdae024d4ebe3c93c3ea1afe37 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8cbcb48cd2ab4a93956f5392df6c5983 (127.0.98.2:39543): Not found: Tablet not found: d2d209f53f204d5288490f1c35807422
I20260927 12:48:53.161358 15319 leader_election.cc:302] T d2d209f53f204d5288490f1c35807422 P 104cf2fdae024d4ebe3c93c3ea1afe37 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 104cf2fdae024d4ebe3c93c3ea1afe37; no voters: 87c9851a903c4436b895d7da878979c7, 8cbcb48cd2ab4a93956f5392df6c5983
I20260927 12:48:53.161481 15711 raft_consensus.cc:2726] T d2d209f53f204d5288490f1c35807422 P 104cf2fdae024d4ebe3c93c3ea1afe37 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 12:48:53.179165 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:53.178750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37309 (local address 127.0.98.1:54351)
0927 12:48:53.179011 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:53.179068 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":98,"threads_started":1}
I20260927 12:48:53.277504 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:53.277110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37309 (local address 127.0.98.2:58421)
0927 12:48:53.277338 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:53.277399 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":71,"threads_started":1}
I20260927 12:48:53.456818 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:53.456383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37309 (local address 127.0.98.3:58467)
0927 12:48:53.456648 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:53.456715 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":84,"threads_started":1}
I20260927 12:48:54.180464 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:54.180010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37309 (local address 127.0.98.1:37127)
0927 12:48:54.180273 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:54.180365 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":87,"threads_started":1}
I20260927 12:48:54.278761 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:54.278284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37309 (local address 127.0.98.2:57553)
0927 12:48:54.278543 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:54.278603 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":79,"threads_started":1}
I20260927 12:48:54.458034 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:54.457528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37309 (local address 127.0.98.3:58489)
0927 12:48:54.457877 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:54.457939 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":164,"threads_started":1}
I20260927 12:48:55.181779 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:55.181284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37309 (local address 127.0.98.1:46847)
0927 12:48:55.181602 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:55.181672 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":78,"threads_started":1}
I20260927 12:48:55.279845 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:55.279456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37309 (local address 127.0.98.2:38709)
0927 12:48:55.279696 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:55.279753 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":72,"threads_started":1}
I20260927 12:48:55.459250 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:55.458829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37309 (local address 127.0.98.3:44395)
0927 12:48:55.459092 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:55.459154 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":72,"threads_started":1}
I20260927 12:48:56.182873 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:56.182456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37309 (local address 127.0.98.1:38531)
0927 12:48:56.182696 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:56.182751 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":82,"threads_started":1}
I20260927 12:48:56.273897 15722 raft_consensus.cc:482] T d2d209f53f204d5288490f1c35807422 P 104cf2fdae024d4ebe3c93c3ea1afe37 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:48:56.274070 15722 raft_consensus.cc:504] T d2d209f53f204d5288490f1c35807422 P 104cf2fdae024d4ebe3c93c3ea1afe37 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c9851a903c4436b895d7da878979c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37579 } } peers { permanent_uuid: "8cbcb48cd2ab4a93956f5392df6c5983" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39543 } } peers { permanent_uuid: "104cf2fdae024d4ebe3c93c3ea1afe37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37565 } }
I20260927 12:48:56.274502 15722 leader_election.cc:288] T d2d209f53f204d5288490f1c35807422 P 104cf2fdae024d4ebe3c93c3ea1afe37 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 87c9851a903c4436b895d7da878979c7 (127.0.98.3:37579), 8cbcb48cd2ab4a93956f5392df6c5983 (127.0.98.2:39543)
W20260927 12:48:56.275254 15318 leader_election.cc:341] T d2d209f53f204d5288490f1c35807422 P 104cf2fdae024d4ebe3c93c3ea1afe37 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 87c9851a903c4436b895d7da878979c7 (127.0.98.3:37579): Not found: Tablet not found: d2d209f53f204d5288490f1c35807422
W20260927 12:48:56.275367 15319 leader_election.cc:341] T d2d209f53f204d5288490f1c35807422 P 104cf2fdae024d4ebe3c93c3ea1afe37 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8cbcb48cd2ab4a93956f5392df6c5983 (127.0.98.2:39543): Not found: Tablet not found: d2d209f53f204d5288490f1c35807422
I20260927 12:48:56.275424 15319 leader_election.cc:302] T d2d209f53f204d5288490f1c35807422 P 104cf2fdae024d4ebe3c93c3ea1afe37 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 104cf2fdae024d4ebe3c93c3ea1afe37; no voters: 87c9851a903c4436b895d7da878979c7, 8cbcb48cd2ab4a93956f5392df6c5983
I20260927 12:48:56.275554 15722 raft_consensus.cc:2726] T d2d209f53f204d5288490f1c35807422 P 104cf2fdae024d4ebe3c93c3ea1afe37 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 12:48:56.280861 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:56.280473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37309 (local address 127.0.98.2:46831)
0927 12:48:56.280717 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:56.280769 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":84,"threads_started":1}
I20260927 12:48:56.460665 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:56.460196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37309 (local address 127.0.98.3:39763)
0927 12:48:56.460487 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:56.460552 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":90,"threads_started":1}
I20260927 12:48:57.184063 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:57.183610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37309 (local address 127.0.98.1:52411)
0927 12:48:57.183905 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:57.183968 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":117,"threads_started":1}
I20260927 12:48:57.282073 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:57.281551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37309 (local address 127.0.98.2:48517)
0927 12:48:57.281886 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:57.281945 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":135,"threads_started":1}
I20260927 12:48:57.461948 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:57.461458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37309 (local address 127.0.98.3:53755)
0927 12:48:57.461779 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:57.461853 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":79,"threads_started":1}
I20260927 12:48:58.185343 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:58.184885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37309 (local address 127.0.98.1:51259)
0927 12:48:58.185184 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:58.185247 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":76,"threads_started":1}
I20260927 12:48:58.283371 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:58.282907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37309 (local address 127.0.98.2:57219)
0927 12:48:58.283208 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:58.283273 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":105,"threads_started":1}
I20260927 12:48:58.463194 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:58.462806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37309 (local address 127.0.98.3:32787)
0927 12:48:58.463042 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:58.463101 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":74,"threads_started":1}
I20260927 12:48:59.186730 15731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:59.186302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37309 (local address 127.0.98.1:59283)
0927 12:48:59.186569 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:59.186634 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":77,"threads_started":1}
I20260927 12:48:59.284721 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:59.284251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37309 (local address 127.0.98.2:32947)
0927 12:48:59.284563 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:59.284627 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":93,"threads_started":1}
I20260927 12:48:59.464465 15733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:48:59.464048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37309 (local address 127.0.98.3:33329)
0927 12:48:59.464294 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 12:48:59.464371 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":72,"threads_started":1}
I20260927 12:49:00.188195 15734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:00.187746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37309 (local address 127.0.98.1:35545)
0927 12:49:00.188031 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:00.188099 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":80,"threads_started":1}
I20260927 12:49:00.286134 15735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:00.285632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37309 (local address 127.0.98.2:37407)
0927 12:49:00.285930 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:00.285998 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":78,"threads_started":1}
I20260927 12:49:00.465890 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:00.465396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37309 (local address 127.0.98.3:53761)
0927 12:49:00.465719 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:00.465791 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":81,"threads_started":1}
I20260927 12:49:00.643993 15737 raft_consensus.cc:482] T d2d209f53f204d5288490f1c35807422 P 104cf2fdae024d4ebe3c93c3ea1afe37 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:49:00.644122 15737 raft_consensus.cc:504] T d2d209f53f204d5288490f1c35807422 P 104cf2fdae024d4ebe3c93c3ea1afe37 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c9851a903c4436b895d7da878979c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37579 } } peers { permanent_uuid: "8cbcb48cd2ab4a93956f5392df6c5983" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39543 } } peers { permanent_uuid: "104cf2fdae024d4ebe3c93c3ea1afe37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37565 } }
I20260927 12:49:00.644513 15737 leader_election.cc:288] T d2d209f53f204d5288490f1c35807422 P 104cf2fdae024d4ebe3c93c3ea1afe37 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 87c9851a903c4436b895d7da878979c7 (127.0.98.3:37579), 8cbcb48cd2ab4a93956f5392df6c5983 (127.0.98.2:39543)
W20260927 12:49:00.645316 15318 leader_election.cc:341] T d2d209f53f204d5288490f1c35807422 P 104cf2fdae024d4ebe3c93c3ea1afe37 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 87c9851a903c4436b895d7da878979c7 (127.0.98.3:37579): Not found: Tablet not found: d2d209f53f204d5288490f1c35807422
W20260927 12:49:00.645434 15319 leader_election.cc:341] T d2d209f53f204d5288490f1c35807422 P 104cf2fdae024d4ebe3c93c3ea1afe37 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8cbcb48cd2ab4a93956f5392df6c5983 (127.0.98.2:39543): Not found: Tablet not found: d2d209f53f204d5288490f1c35807422
I20260927 12:49:00.645519 15319 leader_election.cc:302] T d2d209f53f204d5288490f1c35807422 P 104cf2fdae024d4ebe3c93c3ea1afe37 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 104cf2fdae024d4ebe3c93c3ea1afe37; no voters: 87c9851a903c4436b895d7da878979c7, 8cbcb48cd2ab4a93956f5392df6c5983
I20260927 12:49:00.645656 15737 raft_consensus.cc:2726] T d2d209f53f204d5288490f1c35807422 P 104cf2fdae024d4ebe3c93c3ea1afe37 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 12:49:01.189428 15738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:01.189013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37309 (local address 127.0.98.1:33615)
0927 12:49:01.189266 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:01.189331 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":74,"threads_started":1}
I20260927 12:49:01.287483 15739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:01.287030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37309 (local address 127.0.98.2:60383)
0927 12:49:01.287319 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:01.287385 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":111,"threads_started":1}
I20260927 12:49:01.467100 15740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:01.466670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37309 (local address 127.0.98.3:57785)
0927 12:49:01.466936 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:01.466994 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":79,"threads_started":1}
I20260927 12:49:02.190694 15741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:02.190241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37309 (local address 127.0.98.1:43757)
0927 12:49:02.190522 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:02.190586 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":84,"threads_started":1}
I20260927 12:49:02.288779 15742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:02.288252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37309 (local address 127.0.98.2:54865)
0927 12:49:02.288530 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:02.288614 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":78,"threads_started":1}
I20260927 12:49:02.468767 15743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:02.468079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37309 (local address 127.0.98.3:34111)
0927 12:49:02.468504 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:02.468585 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":101,"threads_started":1}
I20260927 12:49:03.191946 15744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:03.191478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37309 (local address 127.0.98.1:51071)
0927 12:49:03.191778 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:03.191848 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":88,"threads_started":1}
I20260927 12:49:03.290166 15745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:03.289615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37309 (local address 127.0.98.2:58915)
0927 12:49:03.289967 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:03.290032 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":77,"threads_started":1}
I20260927 12:49:03.470130 15746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:03.469644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37309 (local address 127.0.98.3:46269)
0927 12:49:03.469923 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:03.469980 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":91,"threads_started":1}
I20260927 12:49:03.934433 15747 raft_consensus.cc:482] T d2d209f53f204d5288490f1c35807422 P 104cf2fdae024d4ebe3c93c3ea1afe37 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:49:03.934567 15747 raft_consensus.cc:504] T d2d209f53f204d5288490f1c35807422 P 104cf2fdae024d4ebe3c93c3ea1afe37 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87c9851a903c4436b895d7da878979c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37579 } } peers { permanent_uuid: "8cbcb48cd2ab4a93956f5392df6c5983" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39543 } } peers { permanent_uuid: "104cf2fdae024d4ebe3c93c3ea1afe37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37565 } }
I20260927 12:49:03.934986 15747 leader_election.cc:288] T d2d209f53f204d5288490f1c35807422 P 104cf2fdae024d4ebe3c93c3ea1afe37 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 87c9851a903c4436b895d7da878979c7 (127.0.98.3:37579), 8cbcb48cd2ab4a93956f5392df6c5983 (127.0.98.2:39543)
W20260927 12:49:03.935693 15318 leader_election.cc:341] T d2d209f53f204d5288490f1c35807422 P 104cf2fdae024d4ebe3c93c3ea1afe37 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 87c9851a903c4436b895d7da878979c7 (127.0.98.3:37579): Not found: Tablet not found: d2d209f53f204d5288490f1c35807422
W20260927 12:49:03.935874 15319 leader_election.cc:341] T d2d209f53f204d5288490f1c35807422 P 104cf2fdae024d4ebe3c93c3ea1afe37 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8cbcb48cd2ab4a93956f5392df6c5983 (127.0.98.2:39543): Not found: Tablet not found: d2d209f53f204d5288490f1c35807422
I20260927 12:49:03.935959 15319 leader_election.cc:302] T d2d209f53f204d5288490f1c35807422 P 104cf2fdae024d4ebe3c93c3ea1afe37 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 104cf2fdae024d4ebe3c93c3ea1afe37; no voters: 87c9851a903c4436b895d7da878979c7, 8cbcb48cd2ab4a93956f5392df6c5983
I20260927 12:49:03.936090 15747 raft_consensus.cc:2726] T d2d209f53f204d5288490f1c35807422 P 104cf2fdae024d4ebe3c93c3ea1afe37 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 12:49:04.193280 15748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:04.192767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37309 (local address 127.0.98.1:46037)
0927 12:49:04.193092 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:04.193178 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":100,"threads_started":1}
I20260927 12:49:04.291422 15749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:04.290951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37309 (local address 127.0.98.2:38581)
0927 12:49:04.291234 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:04.291324 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":97,"threads_started":1}
I20260927 12:49:04.471243 15750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:04.470769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37309 (local address 127.0.98.3:48769)
0927 12:49:04.471091 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:04.471146 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":116,"threads_started":1}
I20260927 12:49:05.194928 15751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:05.194456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37309 (local address 127.0.98.1:55271)
0927 12:49:05.194752 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:05.194829 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":87,"threads_started":1}
I20260927 12:49:05.292833 15752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:05.292418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37309 (local address 127.0.98.2:56063)
0927 12:49:05.292670 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:05.292731 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":72,"threads_started":1}
I20260927 12:49:05.458648 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 15267
I20260927 12:49:05.463927 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 15434
I20260927 12:49:05.468544 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 15568
2026-09-27T12:49:05Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19273 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:49:05 dist-test-slave-lkn4 krb5kdc[15755](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:49:05 dist-test-slave-lkn4 krb5kdc[15755](info): set up 2 sockets
Sep 27 12:49:05 dist-test-slave-lkn4 krb5kdc[15755](info): commencing operation
krb5kdc: starting...
W20260927 12:49:06.438016 392 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/krb5kdc/test-user.keytab.
Sep 27 12:49:06 dist-test-slave-lkn4 krb5kdc[15755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:49:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:49:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:49:06.562218 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39275
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39275
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:49:06.673064 15768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:49:06.673362 15768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:49:06.673401 15768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:49:06.677466 15768 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 12:49:06.677546 15768 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 12:49:06.677563 15768 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 12:49:06.677577 15768 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:49:06.677594 15768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:49:06.677611 15768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:49:06.677625 15768 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:49:06.677639 15768 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:49:06.683030 15768 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39275
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:49:06.684134 15768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:49:06.685555 15768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:49:06.696669 15774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:49:06.696669 15773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:49:06.696758 15768 server_base.cc:1034] running on GCE node
W20260927 12:49:06.696698 15776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:49:06.697578 15768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:49:06.698870 15768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:49:06.700040 15768 hybrid_clock.cc:648] HybridClock initialized: now 1790513346700017 us; error 45 us; skew 500 ppm
Sep 27 12:49:06 dist-test-slave-lkn4 krb5kdc[15755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513346, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:49:06.702920 15768 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:49:06.703883 15768 webserver.cc:460] Webserver started at http://127.0.98.62:41215/ using document root <none> and password file <none>
I20260927 12:49:06.704428 15768 fs_manager.cc:359] Metadata directory not provided
I20260927 12:49:06.704487 15768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:49:06.704658 15768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:49:06.706748 15768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/master-0/data/instance:
uuid: "9d18a9fee0b24a12961f68eded69434b"
format_stamp: "Formatted at 2026-09-27 12:49:06 on dist-test-slave-lkn4"
server_key: "c4825e7832d512a9cac63866ca115c47"
server_key_iv: "bb5f8489d48d6de6818701151190b796"
server_key_version: "encryptionkey@0"
I20260927 12:49:06.707260 15768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/master-0/wal/instance:
uuid: "9d18a9fee0b24a12961f68eded69434b"
format_stamp: "Formatted at 2026-09-27 12:49:06 on dist-test-slave-lkn4"
server_key: "c4825e7832d512a9cac63866ca115c47"
server_key_iv: "bb5f8489d48d6de6818701151190b796"
server_key_version: "encryptionkey@0"
I20260927 12:49:06.710947 15768 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 12:49:06.713397 15783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:49:06.714689 15768 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:49:06.714812 15768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/master-0/wal
uuid: "9d18a9fee0b24a12961f68eded69434b"
format_stamp: "Formatted at 2026-09-27 12:49:06 on dist-test-slave-lkn4"
server_key: "c4825e7832d512a9cac63866ca115c47"
server_key_iv: "bb5f8489d48d6de6818701151190b796"
server_key_version: "encryptionkey@0"
I20260927 12:49:06.714932 15768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:49:06.726413 15768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:49:06.729524 15768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:49:06.729681 15768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:49:06.736392 15768 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39275
I20260927 12:49:06.736687 15835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39275 every 8 connection(s)
I20260927 12:49:06.737360 15768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:49:06.738531 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 15768
I20260927 12:49:06.738633 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/master-0/wal/instance
I20260927 12:49:06.738910 392 external_mini_cluster.cc:1428] Setting key eea8745218ff3883e0ec124ce03b766d
I20260927 12:49:06.740526 15836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:49:06.747253 15836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d18a9fee0b24a12961f68eded69434b: Bootstrap starting.
Sep 27 12:49:06 dist-test-slave-lkn4 krb5kdc[15755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:49:06.749998 15836 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d18a9fee0b24a12961f68eded69434b: Neither blocks nor log segments found. Creating new log.
I20260927 12:49:06.750993 15836 log.cc:824] T 00000000000000000000000000000000 P 9d18a9fee0b24a12961f68eded69434b: Log is configured to *not* fsync() on all Append() calls
I20260927 12:49:06.753433 15836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d18a9fee0b24a12961f68eded69434b: No bootstrap required, opened a new log
I20260927 12:49:06.756139 15839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:06.740180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39688 (local address 127.0.98.62:39275)
0927 12:49:06.740613 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0927 12:49:06.740632 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 12:49:06.740675 (+ 43us) server_negotiation.cc:407] Connection header received
0927 12:49:06.741524 (+ 849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:49:06.741546 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:49:06.741962 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:49:06.742360 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:49:06.743015 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:49:06.744517 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:49:06.745322 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:49:06.746006 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:49:06.746197 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:49:06.748832 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:49:06.748840 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:49:06.748857 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:49:06.748892 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:49:06.751533 (+ 2641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:49:06.752118 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:49:06.752125 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:49:06.752132 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:49:06.752439 (+ 307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:49:06.752818 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:49:06.752822 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:49:06.752830 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:49:06.753419 (+ 589us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:49:06.754437 (+ 1018us) server_negotiation.cc:1035] Waiting for connection context
0927 12:49:06.754598 (+ 161us) server_negotiation.cc:299] Negotiation successful
0927 12:49:06.754939 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":121,"threads_started":1}
I20260927 12:49:06.757844 15836 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d18a9fee0b24a12961f68eded69434b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d18a9fee0b24a12961f68eded69434b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39275 } }
I20260927 12:49:06.758174 15836 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d18a9fee0b24a12961f68eded69434b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:49:06.758229 15836 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d18a9fee0b24a12961f68eded69434b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d18a9fee0b24a12961f68eded69434b, State: Initialized, Role: FOLLOWER
I20260927 12:49:06.758698 15836 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d18a9fee0b24a12961f68eded69434b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d18a9fee0b24a12961f68eded69434b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39275 } }
I20260927 12:49:06.758876 15836 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d18a9fee0b24a12961f68eded69434b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:49:06.758944 15836 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d18a9fee0b24a12961f68eded69434b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:49:06.759040 15836 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d18a9fee0b24a12961f68eded69434b [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:49:06.760242 15836 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d18a9fee0b24a12961f68eded69434b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d18a9fee0b24a12961f68eded69434b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39275 } }
I20260927 12:49:06.760609 15836 leader_election.cc:302] T 00000000000000000000000000000000 P 9d18a9fee0b24a12961f68eded69434b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d18a9fee0b24a12961f68eded69434b; no voters:
I20260927 12:49:06.760936 15836 leader_election.cc:288] T 00000000000000000000000000000000 P 9d18a9fee0b24a12961f68eded69434b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:49:06.761067 15841 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d18a9fee0b24a12961f68eded69434b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:49:06.761291 15841 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d18a9fee0b24a12961f68eded69434b [term 1 LEADER]: Becoming Leader. State: Replica: 9d18a9fee0b24a12961f68eded69434b, State: Running, Role: LEADER
I20260927 12:49:06.761639 15841 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d18a9fee0b24a12961f68eded69434b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d18a9fee0b24a12961f68eded69434b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39275 } }
I20260927 12:49:06.762269 15836 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d18a9fee0b24a12961f68eded69434b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:49:06.763792 15842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d18a9fee0b24a12961f68eded69434b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d18a9fee0b24a12961f68eded69434b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d18a9fee0b24a12961f68eded69434b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39275 } } }
I20260927 12:49:06.763922 15842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d18a9fee0b24a12961f68eded69434b [sys.catalog]: This master's current role is: LEADER
I20260927 12:49:06.764279 15843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d18a9fee0b24a12961f68eded69434b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d18a9fee0b24a12961f68eded69434b. Latest consensus state: current_term: 1 leader_uuid: "9d18a9fee0b24a12961f68eded69434b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d18a9fee0b24a12961f68eded69434b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39275 } } }
I20260927 12:49:06.764511 15843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d18a9fee0b24a12961f68eded69434b [sys.catalog]: This master's current role is: LEADER
W20260927 12:49:06.767639 15856 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9d18a9fee0b24a12961f68eded69434b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 12:49:06.767705 15856 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 12:49:06.767755 15853 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:49:06.768601 15853 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:49:06.774816 15853 catalog_manager.cc:1347] Generated new cluster ID: 2b6932c0b1cd4fbe8b088af649e732c4
I20260927 12:49:06.774950 15853 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:49:06.803296 15853 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:49:06.804507 15853 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:49:06.813678 15853 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d18a9fee0b24a12961f68eded69434b: Generated new TSK 0
I20260927 12:49:06.814549 15853 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 12:49:06.850611 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39275
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:49:06.961010 15862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:49:06.961288 15862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:49:06.961340 15862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:49:06.965600 15862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:49:06.965685 15862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:49:06.965749 15862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:49:06.971339 15862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15862
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:49:06.972532 15862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:49:06.973577 15862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:49:06.983253 15867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:49:06.983260 15870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:49:06.983260 15868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:49:06.983532 15862 server_base.cc:1034] running on GCE node
I20260927 12:49:06.983978 15862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:49:06.984598 15862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:49:06.985767 15862 hybrid_clock.cc:648] HybridClock initialized: now 1790513346985743 us; error 37 us; skew 500 ppm
Sep 27 12:49:06 dist-test-slave-lkn4 krb5kdc[15755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513346, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:49:06.988674 15862 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 12:49:06.989553 15862 webserver.cc:460] Webserver started at http://127.0.98.1:42935/ using document root <none> and password file <none>
I20260927 12:49:06.990099 15862 fs_manager.cc:359] Metadata directory not provided
I20260927 12:49:06.990163 15862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:49:06.990475 15862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:49:06.992689 15862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-0/data/instance:
uuid: "21c7b42976584d3caecaa0ddbea15cca"
format_stamp: "Formatted at 2026-09-27 12:49:06 on dist-test-slave-lkn4"
server_key: "a542cb87bc53ebb5fa30d52303b8b5b8"
server_key_iv: "e8d014fa90fcdc01ce184f39148df7a6"
server_key_version: "encryptionkey@0"
I20260927 12:49:06.993197 15862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-0/wal/instance:
uuid: "21c7b42976584d3caecaa0ddbea15cca"
format_stamp: "Formatted at 2026-09-27 12:49:06 on dist-test-slave-lkn4"
server_key: "a542cb87bc53ebb5fa30d52303b8b5b8"
server_key_iv: "e8d014fa90fcdc01ce184f39148df7a6"
server_key_version: "encryptionkey@0"
I20260927 12:49:06.996897 15862 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 12:49:06.999366 15877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:49:07.000703 15862 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260927 12:49:07.000824 15862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-0/wal
uuid: "21c7b42976584d3caecaa0ddbea15cca"
format_stamp: "Formatted at 2026-09-27 12:49:06 on dist-test-slave-lkn4"
server_key: "a542cb87bc53ebb5fa30d52303b8b5b8"
server_key_iv: "e8d014fa90fcdc01ce184f39148df7a6"
server_key_version: "encryptionkey@0"
I20260927 12:49:07.000932 15862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:49:07.015384 15862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:49:07.018576 15862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:49:07.018751 15862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:49:07.019352 15862 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:49:07.020151 15862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:49:07.020198 15862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:49:07.020257 15862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:49:07.020282 15862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:49:07.028962 15862 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41501
I20260927 12:49:07.028999 15990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41501 every 8 connection(s)
I20260927 12:49:07.029934 15862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:49:07.037451 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 15862
I20260927 12:49:07.037575 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:49:07.037864 392 external_mini_cluster.cc:1428] Setting key 8f68e1ad9679c19fd01aff0929929f92
Sep 27 12:49:07 dist-test-slave-lkn4 krb5kdc[15755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513346, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:49:07.045019 15839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:07.031922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50387 (local address 127.0.98.62:39275)
0927 12:49:07.032084 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 12:49:07.032089 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:49:07.032958 (+ 869us) server_negotiation.cc:407] Connection header received
0927 12:49:07.034227 (+ 1269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:49:07.034230 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:49:07.034281 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:49:07.034356 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:49:07.035540 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:49:07.036089 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:49:07.037806 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:49:07.038169 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:49:07.038282 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:49:07.040912 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:49:07.040916 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:49:07.040919 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:49:07.040950 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:49:07.042477 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:49:07.043125 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:49:07.043127 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:49:07.043129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:49:07.043206 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:49:07.043631 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:49:07.043634 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:49:07.043635 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:49:07.043930 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:49:07.044027 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 12:49:07.044757 (+ 730us) server_negotiation.cc:299] Negotiation successful
0927 12:49:07.044888 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 12:49:07.046317 15993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:07.032292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39275 (local address 127.0.98.1:50387)
0927 12:49:07.032799 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:07.032847 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 12:49:07.033994 (+ 1147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:49:07.034518 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:49:07.034526 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:49:07.034891 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:49:07.035400 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:49:07.035414 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:49:07.036241 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:49:07.036244 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:49:07.037663 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:49:07.037677 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:49:07.038338 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:49:07.038342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:49:07.038548 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:49:07.039176 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:49:07.039196 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:49:07.040722 (+ 1526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:49:07.042595 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:49:07.042602 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:49:07.042631 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:49:07.043014 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:49:07.043294 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:49:07.043297 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:49:07.043299 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:49:07.043507 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:49:07.044056 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:49:07.044064 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:49:07.044430 (+ 366us) client_negotiation.cc:769] Sending connection context
0927 12:49:07.044712 (+ 282us) client_negotiation.cc:240] Negotiation successful
0927 12:49:07.045112 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":105,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 12:49:07.047592 15991 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39275
I20260927 12:49:07.047886 15991 heartbeater.cc:464] Registering TS with master...
I20260927 12:49:07.048409 15991 heartbeater.cc:511] Master 127.0.98.62:39275 requested a full tablet report, sending...
I20260927 12:49:07.049968 15800 ts_manager.cc:194] Registered new tserver with Master: 21c7b42976584d3caecaa0ddbea15cca (127.0.98.1:41501)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 12:49:07.051390 15800 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50387
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 12:49:07.068743 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39275
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:49:07.178262 15996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:49:07.178524 15996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:49:07.178571 15996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:49:07.182576 15996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:49:07.182639 15996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:49:07.182700 15996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:49:07.188222 15996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:49:07.189283 15996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:49:07.190339 15996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:49:07.200131 16001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:49:07.200121 16004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:49:07.200119 16002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:49:07.200364 15996 server_base.cc:1034] running on GCE node
I20260927 12:49:07.200905 15996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:49:07.201555 15996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:49:07.202772 15996 hybrid_clock.cc:648] HybridClock initialized: now 1790513347202726 us; error 59 us; skew 500 ppm
Sep 27 12:49:07 dist-test-slave-lkn4 krb5kdc[15755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513347, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:49:07.205572 15996 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 12:49:07.206465 15996 webserver.cc:460] Webserver started at http://127.0.98.2:43429/ using document root <none> and password file <none>
I20260927 12:49:07.206996 15996 fs_manager.cc:359] Metadata directory not provided
I20260927 12:49:07.207062 15996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:49:07.207273 15996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:49:07.209256 15996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-1/data/instance:
uuid: "eb3f8d8af27245ee986f70c3eba9975a"
format_stamp: "Formatted at 2026-09-27 12:49:07 on dist-test-slave-lkn4"
server_key: "f993f9f3b93280b8b7738b214cfc2938"
server_key_iv: "63b410bbfac9d7d130bb821ba47bd003"
server_key_version: "encryptionkey@0"
I20260927 12:49:07.209789 15996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-1/wal/instance:
uuid: "eb3f8d8af27245ee986f70c3eba9975a"
format_stamp: "Formatted at 2026-09-27 12:49:07 on dist-test-slave-lkn4"
server_key: "f993f9f3b93280b8b7738b214cfc2938"
server_key_iv: "63b410bbfac9d7d130bb821ba47bd003"
server_key_version: "encryptionkey@0"
I20260927 12:49:07.213677 15996 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 12:49:07.216436 16011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:49:07.217830 15996 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 12:49:07.217953 15996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-1/wal
uuid: "eb3f8d8af27245ee986f70c3eba9975a"
format_stamp: "Formatted at 2026-09-27 12:49:07 on dist-test-slave-lkn4"
server_key: "f993f9f3b93280b8b7738b214cfc2938"
server_key_iv: "63b410bbfac9d7d130bb821ba47bd003"
server_key_version: "encryptionkey@0"
I20260927 12:49:07.218091 15996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:49:07.235286 15996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:49:07.238384 15996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:49:07.238564 15996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:49:07.239171 15996 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:49:07.239992 15996 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:49:07.240037 15996 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:49:07.240087 15996 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:49:07.240115 15996 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:49:07.248677 15996 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43565
I20260927 12:49:07.248704 16124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43565 every 8 connection(s)
I20260927 12:49:07.249555 15996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:49:07.254962 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 15996
I20260927 12:49:07.255071 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:49:07.255342 392 external_mini_cluster.cc:1428] Setting key d3b9d3d99318aa929d59a10b66d60312
Sep 27 12:49:07 dist-test-slave-lkn4 krb5kdc[15755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513347, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:49:07.263458 15839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:07.251358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49645 (local address 127.0.98.62:39275)
0927 12:49:07.251533 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 12:49:07.251538 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:49:07.252493 (+ 955us) server_negotiation.cc:407] Connection header received
0927 12:49:07.253729 (+ 1236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:49:07.253732 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:49:07.253783 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:49:07.253861 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:49:07.255143 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:49:07.255703 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:49:07.256836 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:49:07.257052 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:49:07.257138 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:49:07.259562 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:49:07.259565 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:49:07.259567 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:49:07.259591 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:49:07.261125 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:49:07.261687 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:49:07.261690 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:49:07.261692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:49:07.261859 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:49:07.262250 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:49:07.262254 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:49:07.262256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:49:07.262476 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:49:07.262563 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 12:49:07.263159 (+ 596us) server_negotiation.cc:299] Negotiation successful
0927 12:49:07.263291 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 12:49:07.264564 16127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:07.251869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39275 (local address 127.0.98.2:49645)
0927 12:49:07.252325 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:07.252381 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 12:49:07.253500 (+ 1119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:49:07.254024 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:49:07.254031 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:49:07.254432 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:49:07.254989 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:49:07.255004 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:49:07.255832 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:49:07.255835 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:49:07.256722 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:49:07.256731 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:49:07.257189 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:49:07.257193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:49:07.257356 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:49:07.257918 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:49:07.257935 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:49:07.259397 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:49:07.261303 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:49:07.261310 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:49:07.261326 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:49:07.261589 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:49:07.261953 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:49:07.261956 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:49:07.261958 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:49:07.262157 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:49:07.262573 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:49:07.262581 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:49:07.262863 (+ 282us) client_negotiation.cc:769] Sending connection context
0927 12:49:07.263083 (+ 220us) client_negotiation.cc:240] Negotiation successful
0927 12:49:07.263482 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":110,"threads_started":1}
I20260927 12:49:07.265693 16125 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39275
I20260927 12:49:07.265929 16125 heartbeater.cc:464] Registering TS with master...
I20260927 12:49:07.266475 16125 heartbeater.cc:511] Master 127.0.98.62:39275 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 12:49:07.267576 15800 ts_manager.cc:194] Registered new tserver with Master: eb3f8d8af27245ee986f70c3eba9975a (127.0.98.2:43565)
I20260927 12:49:07.268289 15800 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49645
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 12:49:07.281224 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39275
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:49:07.389307 16130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:49:07.389577 16130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:49:07.389627 16130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:49:07.393819 16130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:49:07.393889 16130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:49:07.393952 16130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:49:07.399519 16130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:49:07.400593 16130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:49:07.401602 16130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:49:07.411515 16136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:49:07.411599 16130 server_base.cc:1034] running on GCE node
W20260927 12:49:07.411506 16135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:49:07.411506 16138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:49:07.412134 16130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:49:07.412801 16130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:49:07.413962 16130 hybrid_clock.cc:648] HybridClock initialized: now 1790513347413941 us; error 33 us; skew 500 ppm
Sep 27 12:49:07 dist-test-slave-lkn4 krb5kdc[15755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513347, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:49:07.416795 16130 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 12:49:07.417687 16130 webserver.cc:460] Webserver started at http://127.0.98.3:37617/ using document root <none> and password file <none>
I20260927 12:49:07.418339 16130 fs_manager.cc:359] Metadata directory not provided
I20260927 12:49:07.418409 16130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:49:07.418610 16130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:49:07.420753 16130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-2/data/instance:
uuid: "b81219f0f4124a45914289257fee36fe"
format_stamp: "Formatted at 2026-09-27 12:49:07 on dist-test-slave-lkn4"
server_key: "290953eaa5097417a0b56a4277a86de1"
server_key_iv: "6e470f406f9e23d155c00d2c35852494"
server_key_version: "encryptionkey@0"
I20260927 12:49:07.421350 16130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b81219f0f4124a45914289257fee36fe"
format_stamp: "Formatted at 2026-09-27 12:49:07 on dist-test-slave-lkn4"
server_key: "290953eaa5097417a0b56a4277a86de1"
server_key_iv: "6e470f406f9e23d155c00d2c35852494"
server_key_version: "encryptionkey@0"
I20260927 12:49:07.425324 16130 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 12:49:07.427953 16145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:49:07.429241 16130 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:49:07.429370 16130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-2/wal
uuid: "b81219f0f4124a45914289257fee36fe"
format_stamp: "Formatted at 2026-09-27 12:49:07 on dist-test-slave-lkn4"
server_key: "290953eaa5097417a0b56a4277a86de1"
server_key_iv: "6e470f406f9e23d155c00d2c35852494"
server_key_version: "encryptionkey@0"
I20260927 12:49:07.429483 16130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:49:07.446630 16130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:49:07.449877 16130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:49:07.450103 16130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:49:07.450714 16130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:49:07.451557 16130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:49:07.451607 16130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:49:07.451659 16130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:49:07.451685 16130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:49:07.460126 16130 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43905
I20260927 12:49:07.460156 16258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43905 every 8 connection(s)
I20260927 12:49:07.461021 16130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:49:07.467420 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 16130
I20260927 12:49:07.467554 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:49:07.467932 392 external_mini_cluster.cc:1428] Setting key 032379c08f235e3d8a9f40685d8247cb
Sep 27 12:49:07 dist-test-slave-lkn4 krb5kdc[15755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513347, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:49:07.474634 15839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:07.462845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38643 (local address 127.0.98.62:39275)
0927 12:49:07.463001 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 12:49:07.463006 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:49:07.464015 (+ 1009us) server_negotiation.cc:407] Connection header received
0927 12:49:07.465309 (+ 1294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:49:07.465312 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:49:07.465363 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:49:07.465436 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:49:07.466606 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:49:07.467153 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:49:07.468614 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:49:07.468894 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:49:07.468978 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:49:07.471417 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:49:07.471420 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:49:07.471422 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:49:07.471449 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:49:07.472839 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:49:07.473273 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:49:07.473276 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:49:07.473277 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:49:07.473356 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:49:07.473612 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:49:07.473614 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:49:07.473615 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:49:07.473796 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:49:07.473869 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 12:49:07.474403 (+ 534us) server_negotiation.cc:299] Negotiation successful
0927 12:49:07.474511 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 12:49:07.475471 16261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:07.463335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39275 (local address 127.0.98.3:38643)
0927 12:49:07.463849 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:07.463908 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 12:49:07.465110 (+ 1202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:49:07.465574 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:49:07.465580 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:49:07.465941 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:49:07.466469 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:49:07.466482 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:49:07.467294 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:49:07.467298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:49:07.468508 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:49:07.468517 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:49:07.469029 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:49:07.469033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:49:07.469206 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:49:07.469755 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:49:07.469776 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:49:07.471236 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:49:07.472938 (+ 1702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:49:07.472943 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:49:07.472954 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:49:07.473185 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:49:07.473424 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:49:07.473426 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:49:07.473428 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:49:07.473544 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:49:07.473867 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:49:07.473873 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:49:07.474153 (+ 280us) client_negotiation.cc:769] Sending connection context
0927 12:49:07.474332 (+ 179us) client_negotiation.cc:240] Negotiation successful
0927 12:49:07.474592 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":104,"threads_started":1}
I20260927 12:49:07.476591 16259 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39275
I20260927 12:49:07.476840 16259 heartbeater.cc:464] Registering TS with master...
I20260927 12:49:07.477324 16259 heartbeater.cc:511] Master 127.0.98.62:39275 requested a full tablet report, sending...
I20260927 12:49:07.478354 15800 ts_manager.cc:194] Registered new tserver with Master: b81219f0f4124a45914289257fee36fe (127.0.98.3:43905)
I20260927 12:49:07.478901 15800 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38643
I20260927 12:49:07.482158 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 12:49:07.491246 15839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:07.484100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39698 (local address 127.0.98.62:39275)
0927 12:49:07.484218 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 12:49:07.484221 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:49:07.484380 (+ 159us) server_negotiation.cc:407] Connection header received
0927 12:49:07.484477 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:49:07.484479 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:49:07.484515 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:49:07.484581 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:49:07.485227 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:49:07.485734 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:49:07.486583 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:49:07.486878 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:49:07.486963 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:49:07.488283 (+ 1320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:49:07.488286 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:49:07.488288 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:49:07.488319 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:49:07.489635 (+ 1316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:49:07.490088 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:49:07.490093 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:49:07.490095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:49:07.490187 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:49:07.490428 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:49:07.490431 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:49:07.490433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:49:07.490721 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:49:07.490827 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 12:49:07.491022 (+ 195us) server_negotiation.cc:299] Negotiation successful
0927 12:49:07.491122 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 12:49:07.494109 15800 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39698:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:49:07.497059 15800 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:49:07.511571 16271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:07.505757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43565 (local address 127.0.0.1:56882)
0927 12:49:07.506757 (+ 1000us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:07.506811 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 12:49:07.506945 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:49:07.507593 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:49:07.507599 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:49:07.507620 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:49:07.507772 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:49:07.507778 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:49:07.508808 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:49:07.508811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:49:07.509949 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:49:07.509959 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:49:07.511024 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:49:07.511027 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:49:07.511153 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:49:07.511193 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 12:49:07.511292 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 12:49:07.511415 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":809,"mutex_wait_us":5,"spinlock_wait_cycles":5632,"thread_start_us":108,"threads_started":1}
I20260927 12:49:07.511580 16272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:07.505910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56882 (local address 127.0.98.2:43565)
0927 12:49:07.507086 (+ 1176us) server_negotiation.cc:206] Beginning negotiation
0927 12:49:07.507097 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 12:49:07.507123 (+ 26us) server_negotiation.cc:407] Connection header received
0927 12:49:07.507220 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:49:07.507226 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:49:07.507448 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:49:07.507592 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:49:07.507901 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:49:07.508693 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:49:07.510117 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:49:07.510916 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:49:07.511009 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:49:07.511165 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 12:49:07.511344 (+ 179us) server_negotiation.cc:299] Negotiation successful
0927 12:49:07.511440 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1091,"thread_start_us":134,"threads_started":1}
I20260927 12:49:07.512040 16270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:07.505796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43905 (local address 127.0.0.1:38522)
0927 12:49:07.506792 (+ 996us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:07.506812 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 12:49:07.507120 (+ 308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:49:07.507877 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:49:07.507880 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:49:07.507897 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:49:07.507977 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:49:07.507982 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:49:07.509371 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:49:07.509374 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:49:07.510518 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:49:07.510527 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:49:07.511704 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:49:07.511707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:49:07.511802 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:49:07.511821 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 12:49:07.511880 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 12:49:07.511931 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":948,"mutex_wait_us":173,"spinlock_wait_cycles":2304,"thread_start_us":85,"threads_started":1}
I20260927 12:49:07.512113 16273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:07.506267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41501 (local address 127.0.0.1:56642)
0927 12:49:07.507155 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:07.507166 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 12:49:07.507243 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:49:07.507708 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:49:07.507711 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:49:07.507729 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:49:07.507835 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:49:07.507841 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:49:07.509369 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:49:07.509373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:49:07.510628 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:49:07.510637 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:49:07.511616 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:49:07.511620 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:49:07.511742 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:49:07.511764 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 12:49:07.511821 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 12:49:07.511885 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":833,"mutex_wait_us":61,"thread_start_us":82,"threads_started":1}
I20260927 12:49:07.512655 16274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:07.506302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56642 (local address 127.0.98.1:41501)
0927 12:49:07.506696 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0927 12:49:07.506710 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 12:49:07.507248 (+ 538us) server_negotiation.cc:407] Connection header received
0927 12:49:07.507360 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:49:07.507366 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:49:07.507563 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:49:07.507959 (+ 396us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:49:07.508102 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:49:07.509216 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:49:07.510743 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:49:07.511497 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:49:07.512277 (+ 780us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:49:07.512427 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 12:49:07.512465 (+ 38us) server_negotiation.cc:299] Negotiation successful
0927 12:49:07.512533 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":62,"threads_started":1}
I20260927 12:49:07.512981 16275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:07.506605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38522 (local address 127.0.98.3:43905)
0927 12:49:07.507480 (+ 875us) server_negotiation.cc:206] Beginning negotiation
0927 12:49:07.507491 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 12:49:07.507511 (+ 20us) server_negotiation.cc:407] Connection header received
0927 12:49:07.507576 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:49:07.507580 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:49:07.507750 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:49:07.507891 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:49:07.508054 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:49:07.509237 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:49:07.510629 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:49:07.511412 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:49:07.511513 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:49:07.511652 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 12:49:07.511935 (+ 283us) server_negotiation.cc:299] Negotiation successful
0927 12:49:07.512850 (+ 915us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":767,"thread_start_us":169,"threads_started":1}
I20260927 12:49:07.514989 16059 tablet_service.cc:1467] Processing CreateTablet for tablet 1c03df73f24a4e2fb7ed32c3100f5e6a (DEFAULT_TABLE table=test-table [id=2168132c613a4200bd70d1749a276d77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:49:07.515353 16193 tablet_service.cc:1467] Processing CreateTablet for tablet 1c03df73f24a4e2fb7ed32c3100f5e6a (DEFAULT_TABLE table=test-table [id=2168132c613a4200bd70d1749a276d77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:49:07.515375 15925 tablet_service.cc:1467] Processing CreateTablet for tablet 1c03df73f24a4e2fb7ed32c3100f5e6a (DEFAULT_TABLE table=test-table [id=2168132c613a4200bd70d1749a276d77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:49:07.516132 16193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c03df73f24a4e2fb7ed32c3100f5e6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:49:07.516137 16059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c03df73f24a4e2fb7ed32c3100f5e6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:49:07.516237 15925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c03df73f24a4e2fb7ed32c3100f5e6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:49:07.523003 16276 tablet_bootstrap.cc:492] T 1c03df73f24a4e2fb7ed32c3100f5e6a P b81219f0f4124a45914289257fee36fe: Bootstrap starting.
I20260927 12:49:07.524163 16278 tablet_bootstrap.cc:492] T 1c03df73f24a4e2fb7ed32c3100f5e6a P 21c7b42976584d3caecaa0ddbea15cca: Bootstrap starting.
I20260927 12:49:07.525761 16276 tablet_bootstrap.cc:654] T 1c03df73f24a4e2fb7ed32c3100f5e6a P b81219f0f4124a45914289257fee36fe: Neither blocks nor log segments found. Creating new log.
I20260927 12:49:07.526317 16278 tablet_bootstrap.cc:654] T 1c03df73f24a4e2fb7ed32c3100f5e6a P 21c7b42976584d3caecaa0ddbea15cca: Neither blocks nor log segments found. Creating new log.
I20260927 12:49:07.526398 16277 tablet_bootstrap.cc:492] T 1c03df73f24a4e2fb7ed32c3100f5e6a P eb3f8d8af27245ee986f70c3eba9975a: Bootstrap starting.
I20260927 12:49:07.527072 16276 log.cc:824] T 1c03df73f24a4e2fb7ed32c3100f5e6a P b81219f0f4124a45914289257fee36fe: Log is configured to *not* fsync() on all Append() calls
I20260927 12:49:07.527781 16278 log.cc:824] T 1c03df73f24a4e2fb7ed32c3100f5e6a P 21c7b42976584d3caecaa0ddbea15cca: Log is configured to *not* fsync() on all Append() calls
I20260927 12:49:07.528733 16277 tablet_bootstrap.cc:654] T 1c03df73f24a4e2fb7ed32c3100f5e6a P eb3f8d8af27245ee986f70c3eba9975a: Neither blocks nor log segments found. Creating new log.
I20260927 12:49:07.529248 16276 tablet_bootstrap.cc:492] T 1c03df73f24a4e2fb7ed32c3100f5e6a P b81219f0f4124a45914289257fee36fe: No bootstrap required, opened a new log
I20260927 12:49:07.529394 16277 log.cc:824] T 1c03df73f24a4e2fb7ed32c3100f5e6a P eb3f8d8af27245ee986f70c3eba9975a: Log is configured to *not* fsync() on all Append() calls
I20260927 12:49:07.529443 16276 ts_tablet_manager.cc:1397] T 1c03df73f24a4e2fb7ed32c3100f5e6a P b81219f0f4124a45914289257fee36fe: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260927 12:49:07.529749 16278 tablet_bootstrap.cc:492] T 1c03df73f24a4e2fb7ed32c3100f5e6a P 21c7b42976584d3caecaa0ddbea15cca: No bootstrap required, opened a new log
I20260927 12:49:07.529927 16278 ts_tablet_manager.cc:1397] T 1c03df73f24a4e2fb7ed32c3100f5e6a P 21c7b42976584d3caecaa0ddbea15cca: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 12:49:07.531009 16277 tablet_bootstrap.cc:492] T 1c03df73f24a4e2fb7ed32c3100f5e6a P eb3f8d8af27245ee986f70c3eba9975a: No bootstrap required, opened a new log
I20260927 12:49:07.531212 16277 ts_tablet_manager.cc:1397] T 1c03df73f24a4e2fb7ed32c3100f5e6a P eb3f8d8af27245ee986f70c3eba9975a: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260927 12:49:07.533360 16276 raft_consensus.cc:348] T 1c03df73f24a4e2fb7ed32c3100f5e6a P b81219f0f4124a45914289257fee36fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3f8d8af27245ee986f70c3eba9975a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43565 } } peers { permanent_uuid: "b81219f0f4124a45914289257fee36fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43905 } } peers { permanent_uuid: "21c7b42976584d3caecaa0ddbea15cca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41501 } }
I20260927 12:49:07.533764 16276 raft_consensus.cc:374] T 1c03df73f24a4e2fb7ed32c3100f5e6a P b81219f0f4124a45914289257fee36fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:49:07.533842 16276 raft_consensus.cc:729] T 1c03df73f24a4e2fb7ed32c3100f5e6a P b81219f0f4124a45914289257fee36fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b81219f0f4124a45914289257fee36fe, State: Initialized, Role: FOLLOWER
I20260927 12:49:07.534082 16278 raft_consensus.cc:348] T 1c03df73f24a4e2fb7ed32c3100f5e6a P 21c7b42976584d3caecaa0ddbea15cca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3f8d8af27245ee986f70c3eba9975a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43565 } } peers { permanent_uuid: "b81219f0f4124a45914289257fee36fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43905 } } peers { permanent_uuid: "21c7b42976584d3caecaa0ddbea15cca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41501 } }
I20260927 12:49:07.534335 16278 raft_consensus.cc:374] T 1c03df73f24a4e2fb7ed32c3100f5e6a P 21c7b42976584d3caecaa0ddbea15cca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:49:07.534390 16278 raft_consensus.cc:729] T 1c03df73f24a4e2fb7ed32c3100f5e6a P 21c7b42976584d3caecaa0ddbea15cca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21c7b42976584d3caecaa0ddbea15cca, State: Initialized, Role: FOLLOWER
I20260927 12:49:07.534324 16276 consensus_queue.cc:260] T 1c03df73f24a4e2fb7ed32c3100f5e6a P b81219f0f4124a45914289257fee36fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3f8d8af27245ee986f70c3eba9975a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43565 } } peers { permanent_uuid: "b81219f0f4124a45914289257fee36fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43905 } } peers { permanent_uuid: "21c7b42976584d3caecaa0ddbea15cca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41501 } }
I20260927 12:49:07.534479 16277 raft_consensus.cc:348] T 1c03df73f24a4e2fb7ed32c3100f5e6a P eb3f8d8af27245ee986f70c3eba9975a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3f8d8af27245ee986f70c3eba9975a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43565 } } peers { permanent_uuid: "b81219f0f4124a45914289257fee36fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43905 } } peers { permanent_uuid: "21c7b42976584d3caecaa0ddbea15cca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41501 } }
I20260927 12:49:07.534732 16277 raft_consensus.cc:374] T 1c03df73f24a4e2fb7ed32c3100f5e6a P eb3f8d8af27245ee986f70c3eba9975a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:49:07.534795 16277 raft_consensus.cc:729] T 1c03df73f24a4e2fb7ed32c3100f5e6a P eb3f8d8af27245ee986f70c3eba9975a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb3f8d8af27245ee986f70c3eba9975a, State: Initialized, Role: FOLLOWER
I20260927 12:49:07.534806 16278 consensus_queue.cc:260] T 1c03df73f24a4e2fb7ed32c3100f5e6a P 21c7b42976584d3caecaa0ddbea15cca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3f8d8af27245ee986f70c3eba9975a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43565 } } peers { permanent_uuid: "b81219f0f4124a45914289257fee36fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43905 } } peers { permanent_uuid: "21c7b42976584d3caecaa0ddbea15cca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41501 } }
I20260927 12:49:07.535192 16277 consensus_queue.cc:260] T 1c03df73f24a4e2fb7ed32c3100f5e6a P eb3f8d8af27245ee986f70c3eba9975a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3f8d8af27245ee986f70c3eba9975a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43565 } } peers { permanent_uuid: "b81219f0f4124a45914289257fee36fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43905 } } peers { permanent_uuid: "21c7b42976584d3caecaa0ddbea15cca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41501 } }
I20260927 12:49:07.535775 16278 ts_tablet_manager.cc:1428] T 1c03df73f24a4e2fb7ed32c3100f5e6a P 21c7b42976584d3caecaa0ddbea15cca: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 12:49:07.535852 16125 heartbeater.cc:503] Master 127.0.98.62:39275 was elected leader, sending a full tablet report...
I20260927 12:49:07.536100 16277 ts_tablet_manager.cc:1428] T 1c03df73f24a4e2fb7ed32c3100f5e6a P eb3f8d8af27245ee986f70c3eba9975a: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260927 12:49:07.536401 16259 heartbeater.cc:503] Master 127.0.98.62:39275 was elected leader, sending a full tablet report...
I20260927 12:49:07.536938 15991 heartbeater.cc:503] Master 127.0.98.62:39275 was elected leader, sending a full tablet report...
I20260927 12:49:07.537001 16276 ts_tablet_manager.cc:1428] T 1c03df73f24a4e2fb7ed32c3100f5e6a P b81219f0f4124a45914289257fee36fe: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260927 12:49:07.682999 16282 raft_consensus.cc:482] T 1c03df73f24a4e2fb7ed32c3100f5e6a P b81219f0f4124a45914289257fee36fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:49:07.683221 16282 raft_consensus.cc:504] T 1c03df73f24a4e2fb7ed32c3100f5e6a P b81219f0f4124a45914289257fee36fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3f8d8af27245ee986f70c3eba9975a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43565 } } peers { permanent_uuid: "b81219f0f4124a45914289257fee36fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43905 } } peers { permanent_uuid: "21c7b42976584d3caecaa0ddbea15cca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41501 } }
I20260927 12:49:07.684542 16282 leader_election.cc:288] T 1c03df73f24a4e2fb7ed32c3100f5e6a P b81219f0f4124a45914289257fee36fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb3f8d8af27245ee986f70c3eba9975a (127.0.98.2:43565), 21c7b42976584d3caecaa0ddbea15cca (127.0.98.1:41501)
I20260927 12:49:07.688036 16274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:07.684961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53771 (local address 127.0.98.1:41501)
0927 12:49:07.685073 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 12:49:07.685076 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:49:07.685091 (+ 15us) server_negotiation.cc:407] Connection header received
0927 12:49:07.685144 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:49:07.685146 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:49:07.685185 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:49:07.685268 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:49:07.685545 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:49:07.686108 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:49:07.687137 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:49:07.687531 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:49:07.687621 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:49:07.687686 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 12:49:07.687848 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 12:49:07.687922 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 12:49:07.688037 16261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:07.684837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41501 (local address 127.0.98.3:53771)
0927 12:49:07.684945 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:07.684961 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:49:07.685064 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:49:07.685306 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:49:07.685309 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:49:07.685330 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:49:07.685438 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:49:07.685443 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:49:07.686207 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:49:07.686217 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:49:07.687015 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:49:07.687024 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:49:07.687637 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:49:07.687641 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:49:07.687743 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:49:07.687761 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 12:49:07.687825 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 12:49:07.687894 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 12:49:07.688424 16272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:07.684952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35627 (local address 127.0.98.2:43565)
0927 12:49:07.685072 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 12:49:07.685077 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:49:07.685521 (+ 444us) server_negotiation.cc:407] Connection header received
0927 12:49:07.685614 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:49:07.685616 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:49:07.685671 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:49:07.685733 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:49:07.685975 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:49:07.686480 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:49:07.687454 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:49:07.687877 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:49:07.687967 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:49:07.688024 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 12:49:07.688238 (+ 214us) server_negotiation.cc:299] Negotiation successful
0927 12:49:07.688309 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 12:49:07.688772 16285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:07.684837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43565 (local address 127.0.98.3:35627)
0927 12:49:07.685430 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:07.685440 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 12:49:07.685527 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:49:07.685752 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:49:07.685755 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:49:07.685770 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:49:07.685875 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:49:07.685880 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:49:07.686580 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:49:07.686583 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:49:07.687345 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:49:07.687353 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:49:07.687985 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:49:07.687988 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:49:07.688089 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:49:07.688111 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 12:49:07.688572 (+ 461us) client_negotiation.cc:240] Negotiation successful
0927 12:49:07.688644 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"mutex_wait_us":294,"thread_start_us":112,"threads_started":1}
I20260927 12:49:07.688767 15945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c03df73f24a4e2fb7ed32c3100f5e6a" candidate_uuid: "b81219f0f4124a45914289257fee36fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21c7b42976584d3caecaa0ddbea15cca" is_pre_election: true
I20260927 12:49:07.689096 15945 raft_consensus.cc:2454] T 1c03df73f24a4e2fb7ed32c3100f5e6a P 21c7b42976584d3caecaa0ddbea15cca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b81219f0f4124a45914289257fee36fe in term 0.
I20260927 12:49:07.689451 16079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c03df73f24a4e2fb7ed32c3100f5e6a" candidate_uuid: "b81219f0f4124a45914289257fee36fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb3f8d8af27245ee986f70c3eba9975a" is_pre_election: true
I20260927 12:49:07.689571 16148 leader_election.cc:302] T 1c03df73f24a4e2fb7ed32c3100f5e6a P b81219f0f4124a45914289257fee36fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21c7b42976584d3caecaa0ddbea15cca, b81219f0f4124a45914289257fee36fe; no voters:
I20260927 12:49:07.689756 16079 raft_consensus.cc:2454] T 1c03df73f24a4e2fb7ed32c3100f5e6a P eb3f8d8af27245ee986f70c3eba9975a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b81219f0f4124a45914289257fee36fe in term 0.
I20260927 12:49:07.689922 16282 raft_consensus.cc:2781] T 1c03df73f24a4e2fb7ed32c3100f5e6a P b81219f0f4124a45914289257fee36fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:49:07.690022 16282 raft_consensus.cc:482] T 1c03df73f24a4e2fb7ed32c3100f5e6a P b81219f0f4124a45914289257fee36fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:49:07.690097 16282 raft_consensus.cc:3037] T 1c03df73f24a4e2fb7ed32c3100f5e6a P b81219f0f4124a45914289257fee36fe [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:49:07.691183 16282 raft_consensus.cc:504] T 1c03df73f24a4e2fb7ed32c3100f5e6a P b81219f0f4124a45914289257fee36fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3f8d8af27245ee986f70c3eba9975a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43565 } } peers { permanent_uuid: "b81219f0f4124a45914289257fee36fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43905 } } peers { permanent_uuid: "21c7b42976584d3caecaa0ddbea15cca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41501 } }
I20260927 12:49:07.691529 16282 leader_election.cc:288] T 1c03df73f24a4e2fb7ed32c3100f5e6a P b81219f0f4124a45914289257fee36fe [CANDIDATE]: Term 1 election: Requested vote from peers eb3f8d8af27245ee986f70c3eba9975a (127.0.98.2:43565), 21c7b42976584d3caecaa0ddbea15cca (127.0.98.1:41501)
I20260927 12:49:07.691985 16079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c03df73f24a4e2fb7ed32c3100f5e6a" candidate_uuid: "b81219f0f4124a45914289257fee36fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb3f8d8af27245ee986f70c3eba9975a"
I20260927 12:49:07.692147 16079 raft_consensus.cc:3037] T 1c03df73f24a4e2fb7ed32c3100f5e6a P eb3f8d8af27245ee986f70c3eba9975a [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:49:07.692042 15945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c03df73f24a4e2fb7ed32c3100f5e6a" candidate_uuid: "b81219f0f4124a45914289257fee36fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21c7b42976584d3caecaa0ddbea15cca"
I20260927 12:49:07.692380 15945 raft_consensus.cc:3037] T 1c03df73f24a4e2fb7ed32c3100f5e6a P 21c7b42976584d3caecaa0ddbea15cca [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:49:07.693624 16079 raft_consensus.cc:2454] T 1c03df73f24a4e2fb7ed32c3100f5e6a P eb3f8d8af27245ee986f70c3eba9975a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b81219f0f4124a45914289257fee36fe in term 1.
I20260927 12:49:07.693650 15945 raft_consensus.cc:2454] T 1c03df73f24a4e2fb7ed32c3100f5e6a P 21c7b42976584d3caecaa0ddbea15cca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b81219f0f4124a45914289257fee36fe in term 1.
I20260927 12:49:07.693962 16148 leader_election.cc:302] T 1c03df73f24a4e2fb7ed32c3100f5e6a P b81219f0f4124a45914289257fee36fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21c7b42976584d3caecaa0ddbea15cca, b81219f0f4124a45914289257fee36fe; no voters:
I20260927 12:49:07.694183 16282 raft_consensus.cc:2781] T 1c03df73f24a4e2fb7ed32c3100f5e6a P b81219f0f4124a45914289257fee36fe [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:49:07.694391 16282 raft_consensus.cc:686] T 1c03df73f24a4e2fb7ed32c3100f5e6a P b81219f0f4124a45914289257fee36fe [term 1 LEADER]: Becoming Leader. State: Replica: b81219f0f4124a45914289257fee36fe, State: Running, Role: LEADER
I20260927 12:49:07.694722 16282 consensus_queue.cc:237] T 1c03df73f24a4e2fb7ed32c3100f5e6a P b81219f0f4124a45914289257fee36fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3f8d8af27245ee986f70c3eba9975a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43565 } } peers { permanent_uuid: "b81219f0f4124a45914289257fee36fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43905 } } peers { permanent_uuid: "21c7b42976584d3caecaa0ddbea15cca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41501 } }
I20260927 12:49:07.698179 15800 catalog_manager.cc:5495] T 1c03df73f24a4e2fb7ed32c3100f5e6a P b81219f0f4124a45914289257fee36fe reported cstate change: term changed from 0 to 1, leader changed from <none> to b81219f0f4124a45914289257fee36fe (127.0.98.3). New cstate: current_term: 1 leader_uuid: "b81219f0f4124a45914289257fee36fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3f8d8af27245ee986f70c3eba9975a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b81219f0f4124a45914289257fee36fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21c7b42976584d3caecaa0ddbea15cca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41501 } health_report { overall_health: UNKNOWN } } }
W20260927 12:49:07.712004 16260 tablet.cc:2369] T 1c03df73f24a4e2fb7ed32c3100f5e6a P b81219f0f4124a45914289257fee36fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 12:49:07.750679 16126 tablet.cc:2369] T 1c03df73f24a4e2fb7ed32c3100f5e6a P eb3f8d8af27245ee986f70c3eba9975a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:49:07.752377 16275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:07.748518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38528 (local address 127.0.98.3:43905)
0927 12:49:07.748732 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 12:49:07.748737 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:49:07.748753 (+ 16us) server_negotiation.cc:407] Connection header received
0927 12:49:07.748846 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:49:07.748849 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:49:07.748905 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:49:07.749001 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:49:07.749292 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:49:07.749794 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:49:07.750721 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:49:07.750987 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:49:07.751083 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:49:07.751441 (+ 358us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:49:07.751929 (+ 488us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:49:07.752028 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 12:49:07.752174 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 12:49:07.752234 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 12:49:07.758455 16079 raft_consensus.cc:1261] T 1c03df73f24a4e2fb7ed32c3100f5e6a P eb3f8d8af27245ee986f70c3eba9975a [term 1 FOLLOWER]: Refusing update from remote peer b81219f0f4124a45914289257fee36fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:49:07.758455 15945 raft_consensus.cc:1261] T 1c03df73f24a4e2fb7ed32c3100f5e6a P 21c7b42976584d3caecaa0ddbea15cca [term 1 FOLLOWER]: Refusing update from remote peer b81219f0f4124a45914289257fee36fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:49:07.759146 16286 consensus_queue.cc:1035] T 1c03df73f24a4e2fb7ed32c3100f5e6a P b81219f0f4124a45914289257fee36fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "21c7b42976584d3caecaa0ddbea15cca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:49:07.759352 16282 consensus_queue.cc:1035] T 1c03df73f24a4e2fb7ed32c3100f5e6a P b81219f0f4124a45914289257fee36fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb3f8d8af27245ee986f70c3eba9975a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:49:07.766655 16291 mvcc.cc:205] Tried to move back new op lower bound from 7333942672410398720 to 7333942672160419840. Current Snapshot: MvccSnapshot[applied={T|T < 7333942672410398720}]
I20260927 12:49:07.767706 16289 mvcc.cc:205] Tried to move back new op lower bound from 7333942672410398720 to 7333942672160419840. Current Snapshot: MvccSnapshot[applied={T|T < 7333942672410398720}]
I20260927 12:49:07.769264 16290 mvcc.cc:205] Tried to move back new op lower bound from 7333942672410398720 to 7333942672160419840. Current Snapshot: MvccSnapshot[applied={T|T < 7333942672410398720}]
I20260927 12:49:07.775250 15795 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:49:07.775439 15795 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:49:07.777908 15795 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9d18a9fee0b24a12961f68eded69434b: Sending DeleteTablet for 3 replicas of tablet 1c03df73f24a4e2fb7ed32c3100f5e6a
I20260927 12:49:07.778648 16059 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c03df73f24a4e2fb7ed32c3100f5e6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:49:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56882
I20260927 12:49:07.778784 16193 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c03df73f24a4e2fb7ed32c3100f5e6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:49:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38522
I20260927 12:49:07.779258 16299 tablet_replica.cc:330] T 1c03df73f24a4e2fb7ed32c3100f5e6a P eb3f8d8af27245ee986f70c3eba9975a: stopping tablet replica
I20260927 12:49:07.779261 15925 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c03df73f24a4e2fb7ed32c3100f5e6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:49:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56642
I20260927 12:49:07.779495 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 15862
I20260927 12:49:07.780256 16299 raft_consensus.cc:2229] T 1c03df73f24a4e2fb7ed32c3100f5e6a P eb3f8d8af27245ee986f70c3eba9975a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:49:07.780625 16299 raft_consensus.cc:2258] T 1c03df73f24a4e2fb7ed32c3100f5e6a P eb3f8d8af27245ee986f70c3eba9975a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:49:07.782284 16299 ts_tablet_manager.cc:1905] T 1c03df73f24a4e2fb7ed32c3100f5e6a P eb3f8d8af27245ee986f70c3eba9975a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:49:07.784929 16299 ts_tablet_manager.cc:1918] T 1c03df73f24a4e2fb7ed32c3100f5e6a P eb3f8d8af27245ee986f70c3eba9975a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:49:07.785022 16299 log.cc:1196] T 1c03df73f24a4e2fb7ed32c3100f5e6a P eb3f8d8af27245ee986f70c3eba9975a: Deleting WAL directory at /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-1/wal/wals/1c03df73f24a4e2fb7ed32c3100f5e6a
I20260927 12:49:07.785305 16299 ts_tablet_manager.cc:1939] T 1c03df73f24a4e2fb7ed32c3100f5e6a P eb3f8d8af27245ee986f70c3eba9975a: Deleting consensus metadata
I20260927 12:49:07.786335 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 15996
W20260927 12:49:07.786669 15786 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41501 (error 108)
I20260927 12:49:07.788386 16300 tablet_replica.cc:330] T 1c03df73f24a4e2fb7ed32c3100f5e6a P b81219f0f4124a45914289257fee36fe: stopping tablet replica
I20260927 12:49:07.788722 16300 raft_consensus.cc:2229] T 1c03df73f24a4e2fb7ed32c3100f5e6a P b81219f0f4124a45914289257fee36fe [term 1 LEADER]: Raft consensus shutting down.
I20260927 12:49:07.789141 16300 raft_consensus.cc:2258] T 1c03df73f24a4e2fb7ed32c3100f5e6a P b81219f0f4124a45914289257fee36fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:49:07.790252 16273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:07.789884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41501 (local address 127.0.0.1:56650)
0927 12:49:07.790084 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:07.790159 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260927 12:49:07.790459 15786 catalog_manager.cc:4584] TS 21c7b42976584d3caecaa0ddbea15cca (127.0.98.1:41501): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1c03df73f24a4e2fb7ed32c3100f5e6a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41501: connect: Connection refused (error 111)
I20260927 12:49:07.791172 16300 ts_tablet_manager.cc:1905] T 1c03df73f24a4e2fb7ed32c3100f5e6a P b81219f0f4124a45914289257fee36fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:49:07.792171 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 16130
I20260927 12:49:07.794173 16273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:07.793810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43565 (local address 127.0.0.1:56892)
0927 12:49:07.793965 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:07.794076 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260927 12:49:07.797660 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 15768
I20260927 12:49:07.797823 16273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:07.797520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43905 (local address 127.0.0.1:38536)
0927 12:49:07.797675 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:07.797737 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.3:43905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 12:49:27.803470 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39275
--webserver_interface=127.0.98.62
--webserver_port=41215
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39275
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:49:27.913484 16303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:49:27.913769 16303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:49:27.913825 16303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:49:27.917806 16303 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 12:49:27.917867 16303 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 12:49:27.917888 16303 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 12:49:27.917906 16303 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:49:27.917928 16303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:49:27.917960 16303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:49:27.917989 16303 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:49:27.918012 16303 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:49:27.923326 16303 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39275
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=41215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:49:27.924444 16303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:49:27.925488 16303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:49:27.938133 16310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:49:27.938115 16312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:49:27.938141 16309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:49:27.938261 16303 server_base.cc:1034] running on GCE node
I20260927 12:49:27.939023 16303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:49:27.940181 16303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:49:27.941337 16303 hybrid_clock.cc:648] HybridClock initialized: now 1790513367941321 us; error 35 us; skew 500 ppm
Sep 27 12:49:27 dist-test-slave-lkn4 krb5kdc[15755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513367, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:49:27.944283 16303 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:49:27.945281 16303 webserver.cc:460] Webserver started at http://127.0.98.62:41215/ using document root <none> and password file <none>
I20260927 12:49:27.945856 16303 fs_manager.cc:359] Metadata directory not provided
I20260927 12:49:27.945940 16303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:49:27.949915 16303 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 12:49:27.952196 16319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:49:27.953573 16303 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 12:49:27.953784 16303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/master-0/wal
uuid: "9d18a9fee0b24a12961f68eded69434b"
format_stamp: "Formatted at 2026-09-27 12:49:06 on dist-test-slave-lkn4"
server_key: "c4825e7832d512a9cac63866ca115c47"
server_key_iv: "bb5f8489d48d6de6818701151190b796"
server_key_version: "encryptionkey@0"
I20260927 12:49:27.954507 16303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:49:27.985013 16303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:49:27.988623 16303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:49:27.988826 16303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:49:27.996148 16303 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39275
I20260927 12:49:27.996457 16371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39275 every 8 connection(s)
I20260927 12:49:27.997247 16303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:49:28.000357 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 16303
I20260927 12:49:28.000813 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41501
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:39275
--webserver_port=42935
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 12:49:28.002084 16372 sys_catalog.cc:263] Verifying existing consensus state
I20260927 12:49:28.003926 16372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d18a9fee0b24a12961f68eded69434b: Bootstrap starting.
I20260927 12:49:28.027235 16372 log.cc:824] T 00000000000000000000000000000000 P 9d18a9fee0b24a12961f68eded69434b: Log is configured to *not* fsync() on all Append() calls
I20260927 12:49:28.038069 16372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d18a9fee0b24a12961f68eded69434b: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 12:49:28.038504 16372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d18a9fee0b24a12961f68eded69434b: Bootstrap complete.
I20260927 12:49:28.043910 16372 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d18a9fee0b24a12961f68eded69434b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d18a9fee0b24a12961f68eded69434b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39275 } }
I20260927 12:49:28.044412 16372 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d18a9fee0b24a12961f68eded69434b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d18a9fee0b24a12961f68eded69434b, State: Initialized, Role: FOLLOWER
I20260927 12:49:28.044977 16372 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d18a9fee0b24a12961f68eded69434b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d18a9fee0b24a12961f68eded69434b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39275 } }
I20260927 12:49:28.045118 16372 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d18a9fee0b24a12961f68eded69434b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:49:28.045192 16372 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d18a9fee0b24a12961f68eded69434b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:49:28.045286 16372 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d18a9fee0b24a12961f68eded69434b [term 1 FOLLOWER]: Advancing to term 2
I20260927 12:49:28.047791 16372 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d18a9fee0b24a12961f68eded69434b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d18a9fee0b24a12961f68eded69434b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39275 } }
I20260927 12:49:28.048161 16372 leader_election.cc:302] T 00000000000000000000000000000000 P 9d18a9fee0b24a12961f68eded69434b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d18a9fee0b24a12961f68eded69434b; no voters:
I20260927 12:49:28.048533 16372 leader_election.cc:288] T 00000000000000000000000000000000 P 9d18a9fee0b24a12961f68eded69434b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 12:49:28.048647 16376 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d18a9fee0b24a12961f68eded69434b [term 2 FOLLOWER]: Leader election won for term 2
I20260927 12:49:28.048914 16376 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d18a9fee0b24a12961f68eded69434b [term 2 LEADER]: Becoming Leader. State: Replica: 9d18a9fee0b24a12961f68eded69434b, State: Running, Role: LEADER
I20260927 12:49:28.049248 16376 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d18a9fee0b24a12961f68eded69434b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d18a9fee0b24a12961f68eded69434b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39275 } }
I20260927 12:49:28.049800 16372 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d18a9fee0b24a12961f68eded69434b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:49:28.050678 16378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d18a9fee0b24a12961f68eded69434b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d18a9fee0b24a12961f68eded69434b. Latest consensus state: current_term: 2 leader_uuid: "9d18a9fee0b24a12961f68eded69434b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d18a9fee0b24a12961f68eded69434b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39275 } } }
I20260927 12:49:28.050812 16378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d18a9fee0b24a12961f68eded69434b [sys.catalog]: This master's current role is: LEADER
I20260927 12:49:28.050664 16377 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d18a9fee0b24a12961f68eded69434b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9d18a9fee0b24a12961f68eded69434b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d18a9fee0b24a12961f68eded69434b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39275 } } }
I20260927 12:49:28.050910 16377 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d18a9fee0b24a12961f68eded69434b [sys.catalog]: This master's current role is: LEADER
I20260927 12:49:28.051211 16385 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:49:28.056057 16385 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:49:28.056638 16385 catalog_manager.cc:1259] Loaded cluster ID: 2b6932c0b1cd4fbe8b088af649e732c4
I20260927 12:49:28.056819 16385 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:49:28.059451 16385 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:49:28.060254 16385 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9d18a9fee0b24a12961f68eded69434b: Loaded TSK: 0
I20260927 12:49:28.073899 16385 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d18a9fee0b24a12961f68eded69434b: Generated new TSK 1
I20260927 12:49:28.074234 16385 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 12:49:28.132680 16374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:49:28.132953 16374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:49:28.132997 16374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:49:28.137286 16374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:49:28.137360 16374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:49:28.137426 16374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:49:28.142868 16374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41501
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=42935
--tserver_master_addrs=127.0.98.62:39275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:49:28.143918 16374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:49:28.144939 16374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:49:28.155740 16399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:49:28.155740 16400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:49:28.155859 16402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:49:28.156498 16374 server_base.cc:1034] running on GCE node
I20260927 12:49:28.156906 16374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:49:28.157598 16374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:49:28.158771 16374 hybrid_clock.cc:648] HybridClock initialized: now 1790513368158751 us; error 32 us; skew 500 ppm
Sep 27 12:49:28 dist-test-slave-lkn4 krb5kdc[15755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513368, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:49:28.161739 16374 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 12:49:28.162926 16374 webserver.cc:460] Webserver started at http://127.0.98.1:42935/ using document root <none> and password file <none>
I20260927 12:49:28.163657 16374 fs_manager.cc:359] Metadata directory not provided
I20260927 12:49:28.163767 16374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:49:28.168937 16374 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 12:49:28.171651 16409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:49:28.173131 16374 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 12:49:28.173254 16374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-0/wal
uuid: "21c7b42976584d3caecaa0ddbea15cca"
format_stamp: "Formatted at 2026-09-27 12:49:06 on dist-test-slave-lkn4"
server_key: "a542cb87bc53ebb5fa30d52303b8b5b8"
server_key_iv: "e8d014fa90fcdc01ce184f39148df7a6"
server_key_version: "encryptionkey@0"
I20260927 12:49:28.173911 16374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:49:28.187585 16374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:49:28.191355 16374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:49:28.191541 16374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:49:28.192178 16374 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:49:28.193409 16416 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 12:49:28.197381 16374 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 12:49:28.197451 16374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260927 12:49:28.197511 16374 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 12:49:28.199620 16374 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 12:49:28.199672 16374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260927 12:49:28.199990 16416 tablet_bootstrap.cc:492] T 1c03df73f24a4e2fb7ed32c3100f5e6a P 21c7b42976584d3caecaa0ddbea15cca: Bootstrap starting.
I20260927 12:49:28.209620 16374 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41501
I20260927 12:49:28.209690 16523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41501 every 8 connection(s)
I20260927 12:49:28.210638 16374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:49:28.218148 16416 log.cc:824] T 1c03df73f24a4e2fb7ed32c3100f5e6a P 21c7b42976584d3caecaa0ddbea15cca: Log is configured to *not* fsync() on all Append() calls
I20260927 12:49:28.218319 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 16374
I20260927 12:49:28.218863 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:43565
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:39275
--webserver_port=43429
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 12:49:28 dist-test-slave-lkn4 krb5kdc[15755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513368, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:49:28.225472 16416 tablet_bootstrap.cc:492] T 1c03df73f24a4e2fb7ed32c3100f5e6a P 21c7b42976584d3caecaa0ddbea15cca: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 12:49:28.225806 16416 tablet_bootstrap.cc:492] T 1c03df73f24a4e2fb7ed32c3100f5e6a P 21c7b42976584d3caecaa0ddbea15cca: Bootstrap complete.
I20260927 12:49:28.226727 16416 ts_tablet_manager.cc:1397] T 1c03df73f24a4e2fb7ed32c3100f5e6a P 21c7b42976584d3caecaa0ddbea15cca: Time spent bootstrapping tablet: real 0.027s user 0.011s sys 0.015s
I20260927 12:49:28.228901 16416 raft_consensus.cc:348] T 1c03df73f24a4e2fb7ed32c3100f5e6a P 21c7b42976584d3caecaa0ddbea15cca [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3f8d8af27245ee986f70c3eba9975a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43565 } } peers { permanent_uuid: "b81219f0f4124a45914289257fee36fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43905 } } peers { permanent_uuid: "21c7b42976584d3caecaa0ddbea15cca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41501 } }
I20260927 12:49:28.229261 16527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:28.213017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42187 (local address 127.0.98.62:39275)
0927 12:49:28.213516 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0927 12:49:28.213534 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 12:49:28.213581 (+ 47us) server_negotiation.cc:407] Connection header received
0927 12:49:28.214819 (+ 1238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:49:28.214835 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:49:28.215243 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:49:28.215632 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:49:28.216737 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:49:28.218210 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:49:28.219542 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:49:28.219993 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:49:28.220141 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:49:28.222964 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:49:28.222971 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:49:28.222987 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:49:28.223021 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:49:28.225104 (+ 2083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:49:28.225750 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:49:28.225757 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:49:28.225762 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:49:28.225921 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:49:28.226429 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:49:28.226433 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:49:28.226435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:49:28.226922 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:49:28.227031 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 12:49:28.227948 (+ 917us) server_negotiation.cc:299] Negotiation successful
0927 12:49:28.228281 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":131,"threads_started":1}
I20260927 12:49:28.229312 16416 raft_consensus.cc:729] T 1c03df73f24a4e2fb7ed32c3100f5e6a P 21c7b42976584d3caecaa0ddbea15cca [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21c7b42976584d3caecaa0ddbea15cca, State: Initialized, Role: FOLLOWER
I20260927 12:49:28.229821 16416 consensus_queue.cc:260] T 1c03df73f24a4e2fb7ed32c3100f5e6a P 21c7b42976584d3caecaa0ddbea15cca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3f8d8af27245ee986f70c3eba9975a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43565 } } peers { permanent_uuid: "b81219f0f4124a45914289257fee36fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43905 } } peers { permanent_uuid: "21c7b42976584d3caecaa0ddbea15cca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41501 } }
I20260927 12:49:28.230773 16416 ts_tablet_manager.cc:1428] T 1c03df73f24a4e2fb7ed32c3100f5e6a P 21c7b42976584d3caecaa0ddbea15cca: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260927 12:49:28.231034 16526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:28.212918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39275 (local address 127.0.98.1:42187)
0927 12:49:28.213403 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:28.213455 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 12:49:28.214320 (+ 865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:49:28.215732 (+ 1412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:49:28.215740 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:49:28.216095 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:49:28.216563 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:49:28.216577 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:49:28.218380 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:49:28.218383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:49:28.219428 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:49:28.219439 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:49:28.220125 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:49:28.220129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:49:28.220309 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:49:28.220983 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:49:28.221002 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:49:28.222748 (+ 1746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:49:28.225247 (+ 2499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:49:28.225253 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:49:28.225267 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:49:28.225603 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:49:28.226067 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:49:28.226070 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:49:28.226072 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:49:28.226299 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:49:28.227118 (+ 819us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:49:28.227124 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:49:28.227437 (+ 313us) client_negotiation.cc:769] Sending connection context
0927 12:49:28.229648 (+ 2211us) client_negotiation.cc:240] Negotiation successful
0927 12:49:28.230076 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":119,"threads_started":1}
I20260927 12:49:28.233237 16524 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39275
I20260927 12:49:28.233445 16524 heartbeater.cc:464] Registering TS with master...
I20260927 12:49:28.233899 16524 heartbeater.cc:511] Master 127.0.98.62:39275 requested a full tablet report, sending...
I20260927 12:49:28.236263 16336 ts_manager.cc:194] Registered new tserver with Master: 21c7b42976584d3caecaa0ddbea15cca (127.0.98.1:41501)
I20260927 12:49:28.239814 16336 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42187
I20260927 12:49:28.242515 16524 heartbeater.cc:503] Master 127.0.98.62:39275 was elected leader, sending a full tablet report...
Sep 27 12:49:28 dist-test-slave-lkn4 krb5kdc[15755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513367, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 12:49:28.249830 16532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:28.239280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47706 (local address 127.0.98.1:41501)
0927 12:49:28.239667 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0927 12:49:28.239677 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 12:49:28.240100 (+ 423us) server_negotiation.cc:407] Connection header received
0927 12:49:28.240286 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:49:28.240292 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:49:28.240497 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:49:28.240638 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:49:28.241339 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:49:28.242405 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:49:28.243598 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:49:28.244060 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:49:28.244140 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:49:28.246610 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:49:28.246618 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:49:28.246628 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:49:28.246650 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:49:28.248127 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:49:28.248625 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:49:28.248629 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:49:28.248630 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:49:28.248699 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:49:28.248945 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:49:28.248947 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:49:28.248948 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:49:28.249147 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:49:28.249258 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 12:49:28.249575 (+ 317us) server_negotiation.cc:299] Negotiation successful
0927 12:49:28.249707 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":103,"threads_started":1}
I20260927 12:49:28.249853 16531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:28.239184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41501 (local address 127.0.0.1:47706)
0927 12:49:28.239950 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:28.240000 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 12:49:28.240140 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:49:28.240633 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:49:28.240639 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:49:28.241086 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:49:28.241216 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:49:28.241225 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:49:28.242536 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:49:28.242540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:49:28.243467 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:49:28.243478 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:49:28.244147 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:49:28.244164 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:49:28.244280 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:49:28.244892 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:49:28.244907 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:49:28.246483 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:49:28.248227 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:49:28.248236 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:49:28.248241 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:49:28.248534 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:49:28.248776 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:49:28.248779 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:49:28.248781 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:49:28.248873 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:49:28.249236 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:49:28.249242 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:49:28.249369 (+ 127us) client_negotiation.cc:769] Sending connection context
0927 12:49:28.249507 (+ 138us) client_negotiation.cc:240] Negotiation successful
0927 12:49:28.249689 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":115,"threads_started":1}
I20260927 12:49:28.250906 16458 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c03df73f24a4e2fb7ed32c3100f5e6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:49:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47706
I20260927 12:49:28.251731 16533 tablet_replica.cc:330] stopping tablet replica
I20260927 12:49:28.252017 16533 raft_consensus.cc:2229] T 1c03df73f24a4e2fb7ed32c3100f5e6a P 21c7b42976584d3caecaa0ddbea15cca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:49:28.252250 16533 raft_consensus.cc:2258] T 1c03df73f24a4e2fb7ed32c3100f5e6a P 21c7b42976584d3caecaa0ddbea15cca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:49:28.253185 16533 ts_tablet_manager.cc:1905] T 1c03df73f24a4e2fb7ed32c3100f5e6a P 21c7b42976584d3caecaa0ddbea15cca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:49:28.257009 16533 ts_tablet_manager.cc:1918] T 1c03df73f24a4e2fb7ed32c3100f5e6a P 21c7b42976584d3caecaa0ddbea15cca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:49:28.257094 16533 log.cc:1196] T 1c03df73f24a4e2fb7ed32c3100f5e6a P 21c7b42976584d3caecaa0ddbea15cca: Deleting WAL directory at /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-0/wal/wals/1c03df73f24a4e2fb7ed32c3100f5e6a
I20260927 12:49:28.257332 16533 ts_tablet_manager.cc:1939] T 1c03df73f24a4e2fb7ed32c3100f5e6a P 21c7b42976584d3caecaa0ddbea15cca: Deleting consensus metadata
I20260927 12:49:28.259202 16322 catalog_manager.cc:4841] TS 21c7b42976584d3caecaa0ddbea15cca (127.0.98.1:41501): tablet 1c03df73f24a4e2fb7ed32c3100f5e6a (table test-table [id=2168132c613a4200bd70d1749a276d77]) successfully deleted
W20260927 12:49:28.335894 16529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:49:28.336172 16529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:49:28.336232 16529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:49:28.340335 16529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:49:28.340407 16529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:49:28.340489 16529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:49:28.345780 16529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:43565
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=43429
--tserver_master_addrs=127.0.98.62:39275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:49:28.346884 16529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:49:28.347896 16529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:49:28.357671 16539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:49:28.357788 16538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:49:28.357815 16529 server_base.cc:1034] running on GCE node
W20260927 12:49:28.358026 16541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:49:28.358438 16529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:49:28.359186 16529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:49:28.360352 16529 hybrid_clock.cc:648] HybridClock initialized: now 1790513368360329 us; error 37 us; skew 500 ppm
Sep 27 12:49:28 dist-test-slave-lkn4 krb5kdc[15755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513368, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:49:28.363111 16529 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 12:49:28.364001 16529 webserver.cc:460] Webserver started at http://127.0.98.2:43429/ using document root <none> and password file <none>
I20260927 12:49:28.364535 16529 fs_manager.cc:359] Metadata directory not provided
I20260927 12:49:28.364615 16529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:49:28.368528 16529 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 12:49:28.370591 16548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:49:28.371866 16529 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:49:28.371973 16529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-1/wal
uuid: "eb3f8d8af27245ee986f70c3eba9975a"
format_stamp: "Formatted at 2026-09-27 12:49:07 on dist-test-slave-lkn4"
server_key: "f993f9f3b93280b8b7738b214cfc2938"
server_key_iv: "63b410bbfac9d7d130bb821ba47bd003"
server_key_version: "encryptionkey@0"
I20260927 12:49:28.372507 16529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:49:28.393399 16529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:49:28.396605 16529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:49:28.396785 16529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:49:28.397377 16529 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:49:28.398228 16529 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:49:28.398276 16529 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:49:28.398332 16529 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:49:28.398360 16529 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:49:28.406886 16529 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43565
I20260927 12:49:28.406919 16661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43565 every 8 connection(s)
I20260927 12:49:28.407840 16529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:49:28.416158 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 16529
I20260927 12:49:28.416625 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:43905
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:39275
--webserver_port=37617
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 12:49:28 dist-test-slave-lkn4 krb5kdc[15755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513368, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:49:28.421552 16527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:28.409627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41569 (local address 127.0.98.62:39275)
0927 12:49:28.409788 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 12:49:28.409791 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:49:28.410648 (+ 857us) server_negotiation.cc:407] Connection header received
0927 12:49:28.411950 (+ 1302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:49:28.411953 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:49:28.412016 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:49:28.412088 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:49:28.413286 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:49:28.413822 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:49:28.414939 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:49:28.415164 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:49:28.415252 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:49:28.417730 (+ 2478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:49:28.417735 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:49:28.417738 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:49:28.417775 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:49:28.419238 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:49:28.419856 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:49:28.419859 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:49:28.419860 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:49:28.419942 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:49:28.420336 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:49:28.420338 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:49:28.420340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:49:28.420537 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:49:28.420624 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 12:49:28.421299 (+ 675us) server_negotiation.cc:299] Negotiation successful
0927 12:49:28.421425 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 12:49:28.422830 16664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:28.409994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39275 (local address 127.0.98.2:41569)
0927 12:49:28.410473 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:28.410518 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 12:49:28.411737 (+ 1219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:49:28.412278 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:49:28.412285 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:49:28.412666 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:49:28.413137 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:49:28.413151 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:49:28.413935 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:49:28.413937 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:49:28.414822 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:49:28.414832 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:49:28.415299 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:49:28.415303 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:49:28.415462 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:49:28.416030 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:49:28.416046 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:49:28.417544 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:49:28.419362 (+ 1818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:49:28.419368 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:49:28.419383 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:49:28.419721 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:49:28.420031 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:49:28.420034 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:49:28.420037 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:49:28.420221 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:49:28.420640 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:49:28.420647 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:49:28.421008 (+ 361us) client_negotiation.cc:769] Sending connection context
0927 12:49:28.421244 (+ 236us) client_negotiation.cc:240] Negotiation successful
0927 12:49:28.421626 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":115,"threads_started":1}
I20260927 12:49:28.424342 16662 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39275
I20260927 12:49:28.424631 16662 heartbeater.cc:464] Registering TS with master...
I20260927 12:49:28.425300 16662 heartbeater.cc:511] Master 127.0.98.62:39275 requested a full tablet report, sending...
I20260927 12:49:28.426538 16336 ts_manager.cc:194] Registered new tserver with Master: eb3f8d8af27245ee986f70c3eba9975a (127.0.98.2:43565)
I20260927 12:49:28.427364 16336 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41569
W20260927 12:49:28.529474 16665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:49:28.529778 16665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:49:28.529824 16665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:49:28.534127 16665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:49:28.534196 16665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:49:28.534264 16665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:49:28.539767 16665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:43905
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=37617
--tserver_master_addrs=127.0.98.62:39275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:49:28.540822 16665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:49:28.541805 16665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:49:28.551283 16673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:49:28.551268 16670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:49:28.551537 16671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:49:28.551549 16665 server_base.cc:1034] running on GCE node
I20260927 12:49:28.552114 16665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:49:28.552801 16665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:49:28.553966 16665 hybrid_clock.cc:648] HybridClock initialized: now 1790513368553946 us; error 35 us; skew 500 ppm
Sep 27 12:49:28 dist-test-slave-lkn4 krb5kdc[15755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513368, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:49:28.556674 16665 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 12:49:28.557533 16665 webserver.cc:460] Webserver started at http://127.0.98.3:37617/ using document root <none> and password file <none>
I20260927 12:49:28.558117 16665 fs_manager.cc:359] Metadata directory not provided
I20260927 12:49:28.558202 16665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:49:28.562117 16665 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 12:49:28.564371 16680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:49:28.565727 16665 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:49:28.565842 16665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-2/wal
uuid: "b81219f0f4124a45914289257fee36fe"
format_stamp: "Formatted at 2026-09-27 12:49:07 on dist-test-slave-lkn4"
server_key: "290953eaa5097417a0b56a4277a86de1"
server_key_iv: "6e470f406f9e23d155c00d2c35852494"
server_key_version: "encryptionkey@0"
I20260927 12:49:28.566417 16665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:49:28.588779 16665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:49:28.592172 16665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:49:28.592355 16665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:49:28.592998 16665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:49:28.594285 16687 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 12:49:28.598522 16665 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 12:49:28.598599 16665 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260927 12:49:28.598678 16665 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 12:49:28.600417 16665 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 12:49:28.600466 16665 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260927 12:49:28.600888 16687 tablet_bootstrap.cc:492] T 1c03df73f24a4e2fb7ed32c3100f5e6a P b81219f0f4124a45914289257fee36fe: Bootstrap starting.
I20260927 12:49:28.607718 16687 log.cc:824] T 1c03df73f24a4e2fb7ed32c3100f5e6a P b81219f0f4124a45914289257fee36fe: Log is configured to *not* fsync() on all Append() calls
I20260927 12:49:28.609869 16665 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43905
I20260927 12:49:28.609937 16795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43905 every 8 connection(s)
I20260927 12:49:28.610772 16665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:49:28.613492 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 16665
I20260927 12:49:28.616770 16687 tablet_bootstrap.cc:492] T 1c03df73f24a4e2fb7ed32c3100f5e6a P b81219f0f4124a45914289257fee36fe: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 12:49:28.617184 16687 tablet_bootstrap.cc:492] T 1c03df73f24a4e2fb7ed32c3100f5e6a P b81219f0f4124a45914289257fee36fe: Bootstrap complete.
I20260927 12:49:28.617584 16687 ts_tablet_manager.cc:1397] T 1c03df73f24a4e2fb7ed32c3100f5e6a P b81219f0f4124a45914289257fee36fe: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.004s
I20260927 12:49:28.620579 16687 raft_consensus.cc:348] T 1c03df73f24a4e2fb7ed32c3100f5e6a P b81219f0f4124a45914289257fee36fe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3f8d8af27245ee986f70c3eba9975a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43565 } } peers { permanent_uuid: "b81219f0f4124a45914289257fee36fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43905 } } peers { permanent_uuid: "21c7b42976584d3caecaa0ddbea15cca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41501 } }
I20260927 12:49:28.621132 16687 raft_consensus.cc:729] T 1c03df73f24a4e2fb7ed32c3100f5e6a P b81219f0f4124a45914289257fee36fe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b81219f0f4124a45914289257fee36fe, State: Initialized, Role: FOLLOWER
Sep 27 12:49:28 dist-test-slave-lkn4 krb5kdc[15755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513368, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:49:28.621690 16687 consensus_queue.cc:260] T 1c03df73f24a4e2fb7ed32c3100f5e6a P b81219f0f4124a45914289257fee36fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3f8d8af27245ee986f70c3eba9975a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43565 } } peers { permanent_uuid: "b81219f0f4124a45914289257fee36fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43905 } } peers { permanent_uuid: "21c7b42976584d3caecaa0ddbea15cca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41501 } }
I20260927 12:49:28.622779 16687 ts_tablet_manager.cc:1428] T 1c03df73f24a4e2fb7ed32c3100f5e6a P b81219f0f4124a45914289257fee36fe: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
I20260927 12:49:28.624603 16799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:28.614259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53400 (local address 127.0.98.62:39275)
0927 12:49:28.614487 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0927 12:49:28.614492 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:49:28.615201 (+ 709us) server_negotiation.cc:407] Connection header received
0927 12:49:28.615314 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:49:28.615316 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:49:28.615356 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:49:28.615427 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:49:28.616130 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:49:28.616791 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:49:28.618034 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:49:28.618324 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:49:28.618412 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:49:28.620033 (+ 1621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:49:28.620037 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:49:28.620040 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:49:28.620067 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:49:28.622387 (+ 2320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:49:28.623103 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:49:28.623107 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:49:28.623109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:49:28.623212 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:49:28.623588 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:49:28.623591 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:49:28.623592 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:49:28.623855 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:49:28.624023 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 12:49:28.624273 (+ 250us) server_negotiation.cc:299] Negotiation successful
0927 12:49:28.624414 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"spinlock_wait_cycles":768,"thread_start_us":61,"threads_started":1}
I20260927 12:49:28.626011 16527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:28.612627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51023 (local address 127.0.98.62:39275)
0927 12:49:28.612781 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 12:49:28.612786 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:49:28.613417 (+ 631us) server_negotiation.cc:407] Connection header received
0927 12:49:28.614881 (+ 1464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:49:28.614885 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:49:28.614944 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:49:28.615023 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:49:28.616524 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:49:28.617097 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:49:28.618664 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:49:28.618915 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:49:28.619009 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:49:28.622291 (+ 3282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:49:28.622295 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:49:28.622298 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:49:28.622339 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:49:28.623757 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:49:28.624344 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:49:28.624347 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:49:28.624349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:49:28.624431 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:49:28.624833 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:49:28.624836 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:49:28.624838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:49:28.625017 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:49:28.625089 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 12:49:28.625756 (+ 667us) server_negotiation.cc:299] Negotiation successful
0927 12:49:28.625870 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52,"spinlock_wait_cycles":7424}
I20260927 12:49:28.626883 16798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:28.612775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39275 (local address 127.0.98.3:51023)
0927 12:49:28.613259 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:28.613312 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 12:49:28.614665 (+ 1353us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:49:28.615232 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:49:28.615241 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:49:28.615714 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:49:28.616372 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:49:28.616392 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:49:28.617229 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:49:28.617232 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:49:28.618542 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:49:28.618552 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:49:28.619143 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:49:28.619147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:49:28.619374 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:49:28.620410 (+ 1036us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:49:28.620431 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:49:28.622106 (+ 1675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:49:28.623855 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:49:28.623861 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:49:28.623877 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:49:28.624215 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:49:28.624542 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:49:28.624544 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:49:28.624546 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:49:28.624721 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:49:28.625152 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:49:28.625158 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:49:28.625488 (+ 330us) client_negotiation.cc:769] Sending connection context
0927 12:49:28.625678 (+ 190us) client_negotiation.cc:240] Negotiation successful
0927 12:49:28.626003 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":120,"threads_started":1}
I20260927 12:49:28.627945 16796 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39275
I20260927 12:49:28.628168 16796 heartbeater.cc:464] Registering TS with master...
I20260927 12:49:28.628651 16796 heartbeater.cc:511] Master 127.0.98.62:39275 requested a full tablet report, sending...
I20260927 12:49:28.630290 16336 ts_manager.cc:194] Registered new tserver with Master: b81219f0f4124a45914289257fee36fe (127.0.98.3:43905)
I20260927 12:49:28.631369 16336 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51023
I20260927 12:49:28.633622 16796 heartbeater.cc:503] Master 127.0.98.62:39275 was elected leader, sending a full tablet report...
Sep 27 12:49:28 dist-test-slave-lkn4 krb5kdc[15755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513367, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260927 12:49:28.638849 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 12:49:28.640707 16531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:28.631098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43905 (local address 127.0.0.1:47522)
0927 12:49:28.631272 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:28.631284 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:49:28.631345 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:49:28.632022 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:49:28.632024 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:49:28.632171 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:49:28.632276 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:49:28.632282 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:49:28.633637 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:49:28.633640 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:49:28.634339 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:49:28.634348 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:49:28.634893 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:49:28.634908 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:49:28.635001 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:49:28.635450 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:49:28.635462 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:49:28.636742 (+ 1280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:49:28.639106 (+ 2364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:49:28.639110 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:49:28.639112 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:49:28.639323 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:49:28.639641 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:49:28.639644 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:49:28.639646 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:49:28.639732 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:49:28.640182 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:49:28.640185 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:49:28.640284 (+ 99us) client_negotiation.cc:769] Sending connection context
0927 12:49:28.640372 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 12:49:28.640541 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":1}
I20260927 12:49:28.640795 16802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:28.631259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47522 (local address 127.0.98.3:43905)
0927 12:49:28.631575 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0927 12:49:28.631587 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 12:49:28.631610 (+ 23us) server_negotiation.cc:407] Connection header received
0927 12:49:28.631683 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:49:28.631689 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:49:28.631886 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:49:28.632011 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:49:28.632386 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:49:28.633513 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:49:28.634440 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:49:28.634805 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:49:28.634905 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:49:28.636948 (+ 2043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:49:28.636959 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:49:28.636976 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:49:28.637003 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:49:28.638955 (+ 1952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:49:28.639431 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:49:28.639438 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:49:28.639440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:49:28.639533 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:49:28.639811 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:49:28.639815 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:49:28.639817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:49:28.640084 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:49:28.640205 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 12:49:28.640470 (+ 265us) server_negotiation.cc:299] Negotiation successful
0927 12:49:28.640638 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":235,"thread_start_us":132,"threads_started":1}
I20260927 12:49:28.641819 16729 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c03df73f24a4e2fb7ed32c3100f5e6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:49:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47522
I20260927 12:49:28.642925 16804 tablet_replica.cc:330] stopping tablet replica
I20260927 12:49:28.643273 16804 raft_consensus.cc:2229] T 1c03df73f24a4e2fb7ed32c3100f5e6a P b81219f0f4124a45914289257fee36fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:49:28.643473 16527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:28.639464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53412 (local address 127.0.98.62:39275)
0927 12:49:28.639593 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 12:49:28.639598 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:49:28.639806 (+ 208us) server_negotiation.cc:407] Connection header received
0927 12:49:28.639961 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:49:28.639964 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:49:28.640067 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:49:28.640155 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:49:28.640403 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:49:28.641128 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:49:28.642243 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:49:28.642464 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:49:28.642554 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:49:28.642943 (+ 389us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:49:28.643106 (+ 163us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0927 12:49:28.643361 (+ 255us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53412: token expired
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":45}
W20260927 12:49:28.643527 16527 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53412: token expired
I20260927 12:49:28.643575 16804 raft_consensus.cc:2258] T 1c03df73f24a4e2fb7ed32c3100f5e6a P b81219f0f4124a45914289257fee36fe [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 12:49:28.643913 16264 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:39275)
I20260927 12:49:28.644668 16804 ts_tablet_manager.cc:1905] T 1c03df73f24a4e2fb7ed32c3100f5e6a P b81219f0f4124a45914289257fee36fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:49:28.649394 16804 ts_tablet_manager.cc:1918] T 1c03df73f24a4e2fb7ed32c3100f5e6a P b81219f0f4124a45914289257fee36fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:49:28.649510 16804 log.cc:1196] T 1c03df73f24a4e2fb7ed32c3100f5e6a P b81219f0f4124a45914289257fee36fe: Deleting WAL directory at /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-2/wal/wals/1c03df73f24a4e2fb7ed32c3100f5e6a
I20260927 12:49:28.649816 16804 ts_tablet_manager.cc:1939] T 1c03df73f24a4e2fb7ed32c3100f5e6a P b81219f0f4124a45914289257fee36fe: Deleting consensus metadata
I20260927 12:49:28.652114 16323 catalog_manager.cc:4841] TS b81219f0f4124a45914289257fee36fe (127.0.98.3:43905): tablet 1c03df73f24a4e2fb7ed32c3100f5e6a (table test-table [id=2168132c613a4200bd70d1749a276d77]) successfully deleted
I20260927 12:49:28.652405 16527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:28.644272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53418 (local address 127.0.98.62:39275)
0927 12:49:28.644361 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0927 12:49:28.644363 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0927 12:49:28.644411 (+ 48us) server_negotiation.cc:407] Connection header received
0927 12:49:28.644525 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:49:28.644528 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:49:28.644560 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:49:28.644615 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:49:28.645317 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:49:28.645868 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:49:28.646991 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:49:28.647195 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:49:28.647306 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:49:28.648756 (+ 1450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:49:28.648759 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:49:28.648762 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:49:28.648789 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:49:28.650594 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:49:28.651083 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:49:28.651086 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:49:28.651088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:49:28.651188 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:49:28.651476 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:49:28.651479 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:49:28.651480 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:49:28.651713 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:49:28.651839 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 12:49:28.652070 (+ 231us) server_negotiation.cc:299] Negotiation successful
0927 12:49:28.652182 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 12:49:28.654765 16336 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53418:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:49:28.658111 16336 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:49:28.665212 16729 tablet_service.cc:1467] Processing CreateTablet for tablet 7bde087239f9498892c0c1bacad362a9 (DEFAULT_TABLE table=test-table [id=01c025b5751243f282dcf19dfb1e5d44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:49:28.665212 16458 tablet_service.cc:1467] Processing CreateTablet for tablet 7bde087239f9498892c0c1bacad362a9 (DEFAULT_TABLE table=test-table [id=01c025b5751243f282dcf19dfb1e5d44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:49:28.665800 16729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bde087239f9498892c0c1bacad362a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:49:28.665827 16458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bde087239f9498892c0c1bacad362a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:49:28.668272 16416 tablet_bootstrap.cc:492] T 7bde087239f9498892c0c1bacad362a9 P 21c7b42976584d3caecaa0ddbea15cca: Bootstrap starting.
I20260927 12:49:28.668865 16687 tablet_bootstrap.cc:492] T 7bde087239f9498892c0c1bacad362a9 P b81219f0f4124a45914289257fee36fe: Bootstrap starting.
I20260927 12:49:28.669066 16416 tablet_bootstrap.cc:654] T 7bde087239f9498892c0c1bacad362a9 P 21c7b42976584d3caecaa0ddbea15cca: Neither blocks nor log segments found. Creating new log.
I20260927 12:49:28.669636 16687 tablet_bootstrap.cc:654] T 7bde087239f9498892c0c1bacad362a9 P b81219f0f4124a45914289257fee36fe: Neither blocks nor log segments found. Creating new log.
I20260927 12:49:28.670585 16416 tablet_bootstrap.cc:492] T 7bde087239f9498892c0c1bacad362a9 P 21c7b42976584d3caecaa0ddbea15cca: No bootstrap required, opened a new log
I20260927 12:49:28.670677 16416 ts_tablet_manager.cc:1397] T 7bde087239f9498892c0c1bacad362a9 P 21c7b42976584d3caecaa0ddbea15cca: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260927 12:49:28.671041 16687 tablet_bootstrap.cc:492] T 7bde087239f9498892c0c1bacad362a9 P b81219f0f4124a45914289257fee36fe: No bootstrap required, opened a new log
I20260927 12:49:28.671125 16687 ts_tablet_manager.cc:1397] T 7bde087239f9498892c0c1bacad362a9 P b81219f0f4124a45914289257fee36fe: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 12:49:28.671162 16416 raft_consensus.cc:348] T 7bde087239f9498892c0c1bacad362a9 P 21c7b42976584d3caecaa0ddbea15cca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3f8d8af27245ee986f70c3eba9975a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43565 } } peers { permanent_uuid: "b81219f0f4124a45914289257fee36fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43905 } } peers { permanent_uuid: "21c7b42976584d3caecaa0ddbea15cca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41501 } }
I20260927 12:49:28.671281 16416 raft_consensus.cc:374] T 7bde087239f9498892c0c1bacad362a9 P 21c7b42976584d3caecaa0ddbea15cca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:49:28.671317 16416 raft_consensus.cc:729] T 7bde087239f9498892c0c1bacad362a9 P 21c7b42976584d3caecaa0ddbea15cca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21c7b42976584d3caecaa0ddbea15cca, State: Initialized, Role: FOLLOWER
I20260927 12:49:28.671434 16416 consensus_queue.cc:260] T 7bde087239f9498892c0c1bacad362a9 P 21c7b42976584d3caecaa0ddbea15cca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3f8d8af27245ee986f70c3eba9975a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43565 } } peers { permanent_uuid: "b81219f0f4124a45914289257fee36fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43905 } } peers { permanent_uuid: "21c7b42976584d3caecaa0ddbea15cca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41501 } }
I20260927 12:49:28.671536 16531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:28.664505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43565 (local address 127.0.0.1:55720)
0927 12:49:28.664849 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:28.664865 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:49:28.664949 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:49:28.665917 (+ 968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:49:28.665920 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:49:28.665939 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:49:28.666109 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:49:28.666119 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:49:28.667512 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:49:28.667516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:49:28.668684 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:49:28.668696 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:49:28.671126 (+ 2430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:49:28.671129 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:49:28.671220 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:49:28.671267 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 12:49:28.671332 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 12:49:28.671411 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191}
I20260927 12:49:28.671589 16808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:28.664689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55720 (local address 127.0.98.2:43565)
0927 12:49:28.665391 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0927 12:49:28.665404 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 12:49:28.665434 (+ 30us) server_negotiation.cc:407] Connection header received
0927 12:49:28.665510 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:49:28.665515 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:49:28.665746 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:49:28.665915 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:49:28.666239 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:49:28.667366 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:49:28.670206 (+ 2840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:49:28.670974 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:49:28.671154 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:49:28.671325 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 12:49:28.671377 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 12:49:28.671468 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":114,"threads_started":1}
I20260927 12:49:28.671542 16687 raft_consensus.cc:348] T 7bde087239f9498892c0c1bacad362a9 P b81219f0f4124a45914289257fee36fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3f8d8af27245ee986f70c3eba9975a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43565 } } peers { permanent_uuid: "b81219f0f4124a45914289257fee36fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43905 } } peers { permanent_uuid: "21c7b42976584d3caecaa0ddbea15cca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41501 } }
I20260927 12:49:28.671641 16687 raft_consensus.cc:374] T 7bde087239f9498892c0c1bacad362a9 P b81219f0f4124a45914289257fee36fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:49:28.671687 16687 raft_consensus.cc:729] T 7bde087239f9498892c0c1bacad362a9 P b81219f0f4124a45914289257fee36fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b81219f0f4124a45914289257fee36fe, State: Initialized, Role: FOLLOWER
I20260927 12:49:28.671808 16687 consensus_queue.cc:260] T 7bde087239f9498892c0c1bacad362a9 P b81219f0f4124a45914289257fee36fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3f8d8af27245ee986f70c3eba9975a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43565 } } peers { permanent_uuid: "b81219f0f4124a45914289257fee36fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43905 } } peers { permanent_uuid: "21c7b42976584d3caecaa0ddbea15cca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41501 } }
I20260927 12:49:28.672060 16687 ts_tablet_manager.cc:1428] T 7bde087239f9498892c0c1bacad362a9 P b81219f0f4124a45914289257fee36fe: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 12:49:28.672389 16416 ts_tablet_manager.cc:1428] T 7bde087239f9498892c0c1bacad362a9 P 21c7b42976584d3caecaa0ddbea15cca: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260927 12:49:28.675225 16596 tablet_service.cc:1467] Processing CreateTablet for tablet 7bde087239f9498892c0c1bacad362a9 (DEFAULT_TABLE table=test-table [id=01c025b5751243f282dcf19dfb1e5d44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:49:28.676214 16596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bde087239f9498892c0c1bacad362a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:49:28.683212 16811 tablet_bootstrap.cc:492] T 7bde087239f9498892c0c1bacad362a9 P eb3f8d8af27245ee986f70c3eba9975a: Bootstrap starting.
I20260927 12:49:28.685940 16811 tablet_bootstrap.cc:654] T 7bde087239f9498892c0c1bacad362a9 P eb3f8d8af27245ee986f70c3eba9975a: Neither blocks nor log segments found. Creating new log.
I20260927 12:49:28.686864 16811 log.cc:824] T 7bde087239f9498892c0c1bacad362a9 P eb3f8d8af27245ee986f70c3eba9975a: Log is configured to *not* fsync() on all Append() calls
I20260927 12:49:28.689133 16811 tablet_bootstrap.cc:492] T 7bde087239f9498892c0c1bacad362a9 P eb3f8d8af27245ee986f70c3eba9975a: No bootstrap required, opened a new log
I20260927 12:49:28.689342 16811 ts_tablet_manager.cc:1397] T 7bde087239f9498892c0c1bacad362a9 P eb3f8d8af27245ee986f70c3eba9975a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 12:49:28.692376 16811 raft_consensus.cc:348] T 7bde087239f9498892c0c1bacad362a9 P eb3f8d8af27245ee986f70c3eba9975a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3f8d8af27245ee986f70c3eba9975a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43565 } } peers { permanent_uuid: "b81219f0f4124a45914289257fee36fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43905 } } peers { permanent_uuid: "21c7b42976584d3caecaa0ddbea15cca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41501 } }
I20260927 12:49:28.692625 16811 raft_consensus.cc:374] T 7bde087239f9498892c0c1bacad362a9 P eb3f8d8af27245ee986f70c3eba9975a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:49:28.692668 16811 raft_consensus.cc:729] T 7bde087239f9498892c0c1bacad362a9 P eb3f8d8af27245ee986f70c3eba9975a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb3f8d8af27245ee986f70c3eba9975a, State: Initialized, Role: FOLLOWER
I20260927 12:49:28.693081 16811 consensus_queue.cc:260] T 7bde087239f9498892c0c1bacad362a9 P eb3f8d8af27245ee986f70c3eba9975a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3f8d8af27245ee986f70c3eba9975a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43565 } } peers { permanent_uuid: "b81219f0f4124a45914289257fee36fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43905 } } peers { permanent_uuid: "21c7b42976584d3caecaa0ddbea15cca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41501 } }
I20260927 12:49:28.693750 16662 heartbeater.cc:503] Master 127.0.98.62:39275 was elected leader, sending a full tablet report...
I20260927 12:49:28.693935 16811 ts_tablet_manager.cc:1428] T 7bde087239f9498892c0c1bacad362a9 P eb3f8d8af27245ee986f70c3eba9975a: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260927 12:49:28.711731 16525 tablet.cc:2369] T 7bde087239f9498892c0c1bacad362a9 P 21c7b42976584d3caecaa0ddbea15cca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:49:28.839766 16813 raft_consensus.cc:482] T 7bde087239f9498892c0c1bacad362a9 P eb3f8d8af27245ee986f70c3eba9975a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:49:28.839975 16813 raft_consensus.cc:504] T 7bde087239f9498892c0c1bacad362a9 P eb3f8d8af27245ee986f70c3eba9975a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3f8d8af27245ee986f70c3eba9975a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43565 } } peers { permanent_uuid: "b81219f0f4124a45914289257fee36fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43905 } } peers { permanent_uuid: "21c7b42976584d3caecaa0ddbea15cca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41501 } }
I20260927 12:49:28.841202 16813 leader_election.cc:288] T 7bde087239f9498892c0c1bacad362a9 P eb3f8d8af27245ee986f70c3eba9975a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b81219f0f4124a45914289257fee36fe (127.0.98.3:43905), 21c7b42976584d3caecaa0ddbea15cca (127.0.98.1:41501)
I20260927 12:49:28.846441 16802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:28.841607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54451 (local address 127.0.98.3:43905)
0927 12:49:28.841720 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 12:49:28.841725 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:49:28.841863 (+ 138us) server_negotiation.cc:407] Connection header received
0927 12:49:28.841924 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:49:28.841926 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:49:28.841985 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:49:28.842122 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:49:28.842372 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:49:28.843193 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:49:28.845038 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:49:28.845795 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:49:28.845881 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:49:28.846008 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 12:49:28.846213 (+ 205us) server_negotiation.cc:299] Negotiation successful
0927 12:49:28.846294 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 12:49:28.846436 16815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:28.841592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47897 (local address 127.0.98.1:41501)
0927 12:49:28.841843 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0927 12:49:28.841848 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:49:28.841991 (+ 143us) server_negotiation.cc:407] Connection header received
0927 12:49:28.842084 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:49:28.842087 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:49:28.842151 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:49:28.842257 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:49:28.842478 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:49:28.843230 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:49:28.845038 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:49:28.845721 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:49:28.845837 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:49:28.846010 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 12:49:28.846244 (+ 234us) server_negotiation.cc:299] Negotiation successful
0927 12:49:28.846322 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":74,"threads_started":1}
I20260927 12:49:28.846462 16664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:28.841489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43905 (local address 127.0.98.2:54451)
0927 12:49:28.841718 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:28.841732 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:49:28.841832 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:49:28.842129 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:49:28.842131 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:49:28.842145 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:49:28.842260 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:49:28.842267 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:49:28.843328 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:49:28.843332 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:49:28.844917 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:49:28.844923 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:49:28.845894 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:49:28.845897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:49:28.846073 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:49:28.846113 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 12:49:28.846196 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 12:49:28.846304 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":131}
I20260927 12:49:28.846470 16814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:28.841489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41501 (local address 127.0.98.2:47897)
0927 12:49:28.841908 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:28.841921 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:49:28.841994 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:49:28.842270 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:49:28.842272 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:49:28.842286 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:49:28.842388 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:49:28.842393 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:49:28.843328 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:49:28.843332 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:49:28.844900 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:49:28.844913 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:49:28.845867 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:49:28.845871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:49:28.846026 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:49:28.846107 (+ 81us) client_negotiation.cc:769] Sending connection context
0927 12:49:28.846205 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 12:49:28.846305 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"mutex_wait_us":228,"thread_start_us":112,"threads_started":1}
I20260927 12:49:28.847158 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bde087239f9498892c0c1bacad362a9" candidate_uuid: "eb3f8d8af27245ee986f70c3eba9975a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b81219f0f4124a45914289257fee36fe" is_pre_election: true
I20260927 12:49:28.847177 16478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bde087239f9498892c0c1bacad362a9" candidate_uuid: "eb3f8d8af27245ee986f70c3eba9975a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21c7b42976584d3caecaa0ddbea15cca" is_pre_election: true
I20260927 12:49:28.847440 16747 raft_consensus.cc:2454] T 7bde087239f9498892c0c1bacad362a9 P b81219f0f4124a45914289257fee36fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb3f8d8af27245ee986f70c3eba9975a in term 0.
I20260927 12:49:28.847443 16478 raft_consensus.cc:2454] T 7bde087239f9498892c0c1bacad362a9 P 21c7b42976584d3caecaa0ddbea15cca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb3f8d8af27245ee986f70c3eba9975a in term 0.
I20260927 12:49:28.847862 16551 leader_election.cc:302] T 7bde087239f9498892c0c1bacad362a9 P eb3f8d8af27245ee986f70c3eba9975a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21c7b42976584d3caecaa0ddbea15cca, eb3f8d8af27245ee986f70c3eba9975a; no voters:
I20260927 12:49:28.848129 16813 raft_consensus.cc:2781] T 7bde087239f9498892c0c1bacad362a9 P eb3f8d8af27245ee986f70c3eba9975a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:49:28.848222 16813 raft_consensus.cc:482] T 7bde087239f9498892c0c1bacad362a9 P eb3f8d8af27245ee986f70c3eba9975a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:49:28.848270 16813 raft_consensus.cc:3037] T 7bde087239f9498892c0c1bacad362a9 P eb3f8d8af27245ee986f70c3eba9975a [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:49:28.849291 16813 raft_consensus.cc:504] T 7bde087239f9498892c0c1bacad362a9 P eb3f8d8af27245ee986f70c3eba9975a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3f8d8af27245ee986f70c3eba9975a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43565 } } peers { permanent_uuid: "b81219f0f4124a45914289257fee36fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43905 } } peers { permanent_uuid: "21c7b42976584d3caecaa0ddbea15cca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41501 } }
I20260927 12:49:28.849642 16813 leader_election.cc:288] T 7bde087239f9498892c0c1bacad362a9 P eb3f8d8af27245ee986f70c3eba9975a [CANDIDATE]: Term 1 election: Requested vote from peers b81219f0f4124a45914289257fee36fe (127.0.98.3:43905), 21c7b42976584d3caecaa0ddbea15cca (127.0.98.1:41501)
I20260927 12:49:28.849954 16747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bde087239f9498892c0c1bacad362a9" candidate_uuid: "eb3f8d8af27245ee986f70c3eba9975a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b81219f0f4124a45914289257fee36fe"
I20260927 12:49:28.850106 16747 raft_consensus.cc:3037] T 7bde087239f9498892c0c1bacad362a9 P b81219f0f4124a45914289257fee36fe [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:49:28.850081 16478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bde087239f9498892c0c1bacad362a9" candidate_uuid: "eb3f8d8af27245ee986f70c3eba9975a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21c7b42976584d3caecaa0ddbea15cca"
I20260927 12:49:28.850209 16478 raft_consensus.cc:3037] T 7bde087239f9498892c0c1bacad362a9 P 21c7b42976584d3caecaa0ddbea15cca [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:49:28.851192 16747 raft_consensus.cc:2454] T 7bde087239f9498892c0c1bacad362a9 P b81219f0f4124a45914289257fee36fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb3f8d8af27245ee986f70c3eba9975a in term 1.
I20260927 12:49:28.851284 16478 raft_consensus.cc:2454] T 7bde087239f9498892c0c1bacad362a9 P 21c7b42976584d3caecaa0ddbea15cca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb3f8d8af27245ee986f70c3eba9975a in term 1.
I20260927 12:49:28.851454 16552 leader_election.cc:302] T 7bde087239f9498892c0c1bacad362a9 P eb3f8d8af27245ee986f70c3eba9975a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b81219f0f4124a45914289257fee36fe, eb3f8d8af27245ee986f70c3eba9975a; no voters:
I20260927 12:49:28.851634 16813 raft_consensus.cc:2781] T 7bde087239f9498892c0c1bacad362a9 P eb3f8d8af27245ee986f70c3eba9975a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:49:28.851856 16813 raft_consensus.cc:686] T 7bde087239f9498892c0c1bacad362a9 P eb3f8d8af27245ee986f70c3eba9975a [term 1 LEADER]: Becoming Leader. State: Replica: eb3f8d8af27245ee986f70c3eba9975a, State: Running, Role: LEADER
I20260927 12:49:28.852233 16813 consensus_queue.cc:237] T 7bde087239f9498892c0c1bacad362a9 P eb3f8d8af27245ee986f70c3eba9975a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3f8d8af27245ee986f70c3eba9975a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43565 } } peers { permanent_uuid: "b81219f0f4124a45914289257fee36fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43905 } } peers { permanent_uuid: "21c7b42976584d3caecaa0ddbea15cca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41501 } }
I20260927 12:49:28.855568 16336 catalog_manager.cc:5495] T 7bde087239f9498892c0c1bacad362a9 P eb3f8d8af27245ee986f70c3eba9975a reported cstate change: term changed from 0 to 1, leader changed from <none> to eb3f8d8af27245ee986f70c3eba9975a (127.0.98.2). New cstate: current_term: 1 leader_uuid: "eb3f8d8af27245ee986f70c3eba9975a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3f8d8af27245ee986f70c3eba9975a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b81219f0f4124a45914289257fee36fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21c7b42976584d3caecaa0ddbea15cca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41501 } health_report { overall_health: UNKNOWN } } }
W20260927 12:49:28.861660 16797 tablet.cc:2369] T 7bde087239f9498892c0c1bacad362a9 P b81219f0f4124a45914289257fee36fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:49:28.907294 16808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:28.903636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55728 (local address 127.0.98.2:43565)
0927 12:49:28.903763 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 12:49:28.903766 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:49:28.903779 (+ 13us) server_negotiation.cc:407] Connection header received
0927 12:49:28.903895 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:49:28.903897 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:49:28.903953 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:49:28.904041 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:49:28.904265 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:49:28.904845 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:49:28.905720 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:49:28.905931 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:49:28.906020 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:49:28.906377 (+ 357us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:49:28.906819 (+ 442us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:49:28.906915 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 12:49:28.907058 (+ 143us) server_negotiation.cc:299] Negotiation successful
0927 12:49:28.907129 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260927 12:49:28.908890 16663 tablet.cc:2369] T 7bde087239f9498892c0c1bacad362a9 P eb3f8d8af27245ee986f70c3eba9975a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:49:28.913221 16747 raft_consensus.cc:1261] T 7bde087239f9498892c0c1bacad362a9 P b81219f0f4124a45914289257fee36fe [term 1 FOLLOWER]: Refusing update from remote peer eb3f8d8af27245ee986f70c3eba9975a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:49:28.913223 16478 raft_consensus.cc:1261] T 7bde087239f9498892c0c1bacad362a9 P 21c7b42976584d3caecaa0ddbea15cca [term 1 FOLLOWER]: Refusing update from remote peer eb3f8d8af27245ee986f70c3eba9975a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:49:28.913862 16816 consensus_queue.cc:1035] T 7bde087239f9498892c0c1bacad362a9 P eb3f8d8af27245ee986f70c3eba9975a [LEADER]: Connected to new peer: Peer: permanent_uuid: "21c7b42976584d3caecaa0ddbea15cca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:49:28.914032 16813 consensus_queue.cc:1035] T 7bde087239f9498892c0c1bacad362a9 P eb3f8d8af27245ee986f70c3eba9975a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b81219f0f4124a45914289257fee36fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:49:28.920816 16821 mvcc.cc:205] Tried to move back new op lower bound from 7333942759061860352 to 7333942758821584896. Current Snapshot: MvccSnapshot[applied={T|T < 7333942759061860352}]
I20260927 12:49:28.928732 16336 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:49:28.928895 16336 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53418:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:49:28.931368 16336 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9d18a9fee0b24a12961f68eded69434b: Sending DeleteTablet for 3 replicas of tablet 7bde087239f9498892c0c1bacad362a9
I20260927 12:49:28.932083 16729 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bde087239f9498892c0c1bacad362a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:49:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47522
I20260927 12:49:28.932107 16596 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bde087239f9498892c0c1bacad362a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:49:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55720
I20260927 12:49:28.932140 16458 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bde087239f9498892c0c1bacad362a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:49:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47706
I20260927 12:49:28.932273 16804 tablet_replica.cc:330] T 7bde087239f9498892c0c1bacad362a9 P b81219f0f4124a45914289257fee36fe: stopping tablet replica
I20260927 12:49:28.932392 16804 raft_consensus.cc:2229] T 7bde087239f9498892c0c1bacad362a9 P b81219f0f4124a45914289257fee36fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:49:28.932523 16804 raft_consensus.cc:2258] T 7bde087239f9498892c0c1bacad362a9 P b81219f0f4124a45914289257fee36fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:49:28.932621 16830 tablet_replica.cc:330] T 7bde087239f9498892c0c1bacad362a9 P eb3f8d8af27245ee986f70c3eba9975a: stopping tablet replica
I20260927 12:49:28.932935 16830 raft_consensus.cc:2229] T 7bde087239f9498892c0c1bacad362a9 P eb3f8d8af27245ee986f70c3eba9975a [term 1 LEADER]: Raft consensus shutting down.
I20260927 12:49:28.933025 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 16374
I20260927 12:49:28.933348 16830 raft_consensus.cc:2258] T 7bde087239f9498892c0c1bacad362a9 P eb3f8d8af27245ee986f70c3eba9975a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:49:28.933460 16804 ts_tablet_manager.cc:1905] T 7bde087239f9498892c0c1bacad362a9 P b81219f0f4124a45914289257fee36fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:49:28.936039 16804 ts_tablet_manager.cc:1918] T 7bde087239f9498892c0c1bacad362a9 P b81219f0f4124a45914289257fee36fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:49:28.936120 16804 log.cc:1196] T 7bde087239f9498892c0c1bacad362a9 P b81219f0f4124a45914289257fee36fe: Deleting WAL directory at /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-2/wal/wals/7bde087239f9498892c0c1bacad362a9
I20260927 12:49:28.936479 16804 ts_tablet_manager.cc:1939] T 7bde087239f9498892c0c1bacad362a9 P b81219f0f4124a45914289257fee36fe: Deleting consensus metadata
I20260927 12:49:28.937212 16323 catalog_manager.cc:4841] TS b81219f0f4124a45914289257fee36fe (127.0.98.3:43905): tablet 7bde087239f9498892c0c1bacad362a9 (table test-table [id=01c025b5751243f282dcf19dfb1e5d44]) successfully deleted
I20260927 12:49:28.939371 16830 ts_tablet_manager.cc:1905] T 7bde087239f9498892c0c1bacad362a9 P eb3f8d8af27245ee986f70c3eba9975a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 12:49:28.940132 16322 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41501 (error 108)
I20260927 12:49:28.940560 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 16529
I20260927 12:49:28.941354 16531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:28.941062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41501 (local address 127.0.0.1:47718)
0927 12:49:28.941221 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:28.941282 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260927 12:49:28.941526 16322 catalog_manager.cc:4584] TS 21c7b42976584d3caecaa0ddbea15cca (127.0.98.1:41501): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7bde087239f9498892c0c1bacad362a9: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41501: connect: Connection refused (error 111)
I20260927 12:49:28.946552 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 16665
I20260927 12:49:28.947098 16531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:28.946721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43565 (local address 127.0.0.1:55732)
0927 12:49:28.946966 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:28.947015 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:43565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179}
I20260927 12:49:28.951323 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 16303
I20260927 12:49:48.957319 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39275
--webserver_interface=127.0.98.62
--webserver_port=41215
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39275
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:49:49.066903 16832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:49:49.067193 16832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:49:49.067241 16832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:49:49.071317 16832 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 12:49:49.071376 16832 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 12:49:49.071391 16832 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 12:49:49.071405 16832 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:49:49.071421 16832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:49:49.071440 16832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:49:49.071470 16832 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:49:49.071496 16832 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:49:49.076853 16832 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39275
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=41215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:49:49.077936 16832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:49:49.079017 16832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:49:49.087940 16840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:49:49.087981 16837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:49:49.088001 16832 server_base.cc:1034] running on GCE node
W20260927 12:49:49.087965 16838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:49:49.088690 16832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:49:49.089846 16832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:49:49.091030 16832 hybrid_clock.cc:648] HybridClock initialized: now 1790513389091022 us; error 41 us; skew 500 ppm
Sep 27 12:49:49 dist-test-slave-lkn4 krb5kdc[15755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513389, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:49:49.093978 16832 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:49:49.094893 16832 webserver.cc:460] Webserver started at http://127.0.98.62:41215/ using document root <none> and password file <none>
I20260927 12:49:49.095448 16832 fs_manager.cc:359] Metadata directory not provided
I20260927 12:49:49.095515 16832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:49:49.099231 16832 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:49:49.101500 16847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:49:49.102887 16832 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 12:49:49.103004 16832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/master-0/wal
uuid: "9d18a9fee0b24a12961f68eded69434b"
format_stamp: "Formatted at 2026-09-27 12:49:06 on dist-test-slave-lkn4"
server_key: "c4825e7832d512a9cac63866ca115c47"
server_key_iv: "bb5f8489d48d6de6818701151190b796"
server_key_version: "encryptionkey@0"
I20260927 12:49:49.103580 16832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:49:49.120312 16832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:49:49.123502 16832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:49:49.123689 16832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:49:49.130681 16832 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39275
I20260927 12:49:49.130951 16899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39275 every 8 connection(s)
I20260927 12:49:49.131773 16832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:49:49.133991 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 16832
I20260927 12:49:49.134507 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41501
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:39275
--webserver_port=42935
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 12:49:49.137085 16900 sys_catalog.cc:263] Verifying existing consensus state
I20260927 12:49:49.138870 16900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d18a9fee0b24a12961f68eded69434b: Bootstrap starting.
I20260927 12:49:49.156366 16900 log.cc:824] T 00000000000000000000000000000000 P 9d18a9fee0b24a12961f68eded69434b: Log is configured to *not* fsync() on all Append() calls
I20260927 12:49:49.168072 16900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d18a9fee0b24a12961f68eded69434b: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 12:49:49.168458 16900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d18a9fee0b24a12961f68eded69434b: Bootstrap complete.
I20260927 12:49:49.172317 16900 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d18a9fee0b24a12961f68eded69434b [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d18a9fee0b24a12961f68eded69434b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39275 } }
I20260927 12:49:49.172712 16900 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d18a9fee0b24a12961f68eded69434b [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d18a9fee0b24a12961f68eded69434b, State: Initialized, Role: FOLLOWER
I20260927 12:49:49.173194 16900 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d18a9fee0b24a12961f68eded69434b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d18a9fee0b24a12961f68eded69434b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39275 } }
I20260927 12:49:49.173310 16900 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d18a9fee0b24a12961f68eded69434b [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:49:49.173372 16900 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d18a9fee0b24a12961f68eded69434b [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:49:49.173465 16900 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d18a9fee0b24a12961f68eded69434b [term 2 FOLLOWER]: Advancing to term 3
I20260927 12:49:49.175307 16900 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d18a9fee0b24a12961f68eded69434b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d18a9fee0b24a12961f68eded69434b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39275 } }
I20260927 12:49:49.175616 16900 leader_election.cc:302] T 00000000000000000000000000000000 P 9d18a9fee0b24a12961f68eded69434b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d18a9fee0b24a12961f68eded69434b; no voters:
I20260927 12:49:49.175910 16900 leader_election.cc:288] T 00000000000000000000000000000000 P 9d18a9fee0b24a12961f68eded69434b [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 12:49:49.176007 16904 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d18a9fee0b24a12961f68eded69434b [term 3 FOLLOWER]: Leader election won for term 3
I20260927 12:49:49.176189 16904 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d18a9fee0b24a12961f68eded69434b [term 3 LEADER]: Becoming Leader. State: Replica: 9d18a9fee0b24a12961f68eded69434b, State: Running, Role: LEADER
I20260927 12:49:49.176548 16904 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d18a9fee0b24a12961f68eded69434b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d18a9fee0b24a12961f68eded69434b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39275 } }
I20260927 12:49:49.177145 16900 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d18a9fee0b24a12961f68eded69434b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:49:49.177907 16905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d18a9fee0b24a12961f68eded69434b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "9d18a9fee0b24a12961f68eded69434b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d18a9fee0b24a12961f68eded69434b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39275 } } }
I20260927 12:49:49.178032 16905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d18a9fee0b24a12961f68eded69434b [sys.catalog]: This master's current role is: LEADER
I20260927 12:49:49.178431 16906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d18a9fee0b24a12961f68eded69434b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d18a9fee0b24a12961f68eded69434b. Latest consensus state: current_term: 3 leader_uuid: "9d18a9fee0b24a12961f68eded69434b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d18a9fee0b24a12961f68eded69434b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39275 } } }
I20260927 12:49:49.178558 16906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d18a9fee0b24a12961f68eded69434b [sys.catalog]: This master's current role is: LEADER
I20260927 12:49:49.178839 16913 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:49:49.184051 16913 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:49:49.184710 16913 catalog_manager.cc:1259] Loaded cluster ID: 2b6932c0b1cd4fbe8b088af649e732c4
I20260927 12:49:49.184772 16913 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:49:49.187417 16913 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:49:49.188171 16913 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9d18a9fee0b24a12961f68eded69434b: Loaded TSK: 0
I20260927 12:49:49.188261 16913 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9d18a9fee0b24a12961f68eded69434b: Loaded TSK: 1
F20260927 12:49:49.188999 16913 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f21526e39ad google::LogMessage::Fail() at ??:0
@ 0x7f21526e688a google::LogMessage::SendToLog() at ??:0
@ 0x7f21526e34de google::LogMessage::Flush() at ??:0
@ 0x7f21526e4a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f21553a718d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f215539ff0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f215d0487bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f215d02299c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f215d023178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f215d064df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f215d08382b std::function<>::operator()() at ??:0
@ 0x7f215d022b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f215d023f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f215d020579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f215d064640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f21549353d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f2154935c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f21549373f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f2154926512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f2155f6a6db start_thread at ??:0
@ 0x7f21511e161f clone at ??:0
W20260927 12:49:49.280314 16902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:49:49.280683 16902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:49:49.280748 16902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:49:49.285677 16902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:49:49.285758 16902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:49:49.285844 16902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:49:49.291334 16902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41501
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=42935
--tserver_master_addrs=127.0.98.62:39275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16902
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:49:49.292423 16902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:49:49.293452 16902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:49:49.303062 16950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:49:49.303045 16948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:49:49.303066 16947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:49:49.303397 16902 server_base.cc:1034] running on GCE node
I20260927 12:49:49.303802 16902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:49:49.304476 16902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:49:49.305672 16902 hybrid_clock.cc:648] HybridClock initialized: now 1790513389305651 us; error 31 us; skew 500 ppm
Sep 27 12:49:49 dist-test-slave-lkn4 krb5kdc[15755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513389, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:49:49.308418 16902 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 12:49:49.309346 16902 webserver.cc:460] Webserver started at http://127.0.98.1:42935/ using document root <none> and password file <none>
I20260927 12:49:49.309900 16902 fs_manager.cc:359] Metadata directory not provided
I20260927 12:49:49.309963 16902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:49:49.313818 16902 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:49:49.315999 16957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:49:49.317216 16902 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 12:49:49.317322 16902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-0/wal
uuid: "21c7b42976584d3caecaa0ddbea15cca"
format_stamp: "Formatted at 2026-09-27 12:49:06 on dist-test-slave-lkn4"
server_key: "a542cb87bc53ebb5fa30d52303b8b5b8"
server_key_iv: "e8d014fa90fcdc01ce184f39148df7a6"
server_key_version: "encryptionkey@0"
I20260927 12:49:49.317878 16902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:49:49.337128 16902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:49:49.340447 16902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:49:49.340621 16902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:49:49.341228 16902 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:49:49.342351 16964 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 12:49:49.346443 16902 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 12:49:49.346510 16902 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260927 12:49:49.346562 16902 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 12:49:49.348248 16902 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 12:49:49.348312 16902 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260927 12:49:49.348592 16964 tablet_bootstrap.cc:492] T 7bde087239f9498892c0c1bacad362a9 P 21c7b42976584d3caecaa0ddbea15cca: Bootstrap starting.
I20260927 12:49:49.358735 16902 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41501
I20260927 12:49:49.359097 17071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41501 every 8 connection(s)
I20260927 12:49:49.360100 16902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:49:49.361658 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 16902
I20260927 12:49:49.362217 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:43565
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:39275
--webserver_port=43429
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 12:49:49.372340 16964 log.cc:824] T 7bde087239f9498892c0c1bacad362a9 P 21c7b42976584d3caecaa0ddbea15cca: Log is configured to *not* fsync() on all Append() calls
I20260927 12:49:49.381322 16964 tablet_bootstrap.cc:492] T 7bde087239f9498892c0c1bacad362a9 P 21c7b42976584d3caecaa0ddbea15cca: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 12:49:49.381733 16964 tablet_bootstrap.cc:492] T 7bde087239f9498892c0c1bacad362a9 P 21c7b42976584d3caecaa0ddbea15cca: Bootstrap complete.
I20260927 12:49:49.383087 16964 ts_tablet_manager.cc:1397] T 7bde087239f9498892c0c1bacad362a9 P 21c7b42976584d3caecaa0ddbea15cca: Time spent bootstrapping tablet: real 0.035s user 0.025s sys 0.007s
I20260927 12:49:49.386371 16964 raft_consensus.cc:348] T 7bde087239f9498892c0c1bacad362a9 P 21c7b42976584d3caecaa0ddbea15cca [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3f8d8af27245ee986f70c3eba9975a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43565 } } peers { permanent_uuid: "b81219f0f4124a45914289257fee36fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43905 } } peers { permanent_uuid: "21c7b42976584d3caecaa0ddbea15cca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41501 } }
I20260927 12:49:49.386919 16964 raft_consensus.cc:729] T 7bde087239f9498892c0c1bacad362a9 P 21c7b42976584d3caecaa0ddbea15cca [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21c7b42976584d3caecaa0ddbea15cca, State: Initialized, Role: FOLLOWER
I20260927 12:49:49.387429 16964 consensus_queue.cc:260] T 7bde087239f9498892c0c1bacad362a9 P 21c7b42976584d3caecaa0ddbea15cca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3f8d8af27245ee986f70c3eba9975a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43565 } } peers { permanent_uuid: "b81219f0f4124a45914289257fee36fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43905 } } peers { permanent_uuid: "21c7b42976584d3caecaa0ddbea15cca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41501 } }
I20260927 12:49:49.388474 16964 ts_tablet_manager.cc:1428] T 7bde087239f9498892c0c1bacad362a9 P 21c7b42976584d3caecaa0ddbea15cca: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260927 12:49:49.398422 17074 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:49:49.362465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39275 (local address 127.0.98.1:36063)
0927 12:49:49.364127 (+ 1662us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:49.364191 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 12:49:49.365317 (+ 1126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:49:49.397424 (+ 32107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39275: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1393,"thread_start_us":165,"threads_started":1}
W20260927 12:49:49.399025 17072 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39275 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:39275: Client connection negotiation failed: client connection to 127.0.98.62:39275: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 12:49:49.399771 17074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:49.399486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39275 (local address 127.0.98.1:34983)
0927 12:49:49.399622 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:49.399700 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 12:49:49.400480 17074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:49.400259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39275 (local address 127.0.98.1:43757)
0927 12:49:49.400369 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:49.400423 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 12:49:49.400692 17072 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 12:49:49.482491 17075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:49:49.482770 17075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:49:49.482818 17075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:49:49.487010 17075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:49:49.487085 17075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:49:49.487149 17075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:49:49.492614 17075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:43565
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=43429
--tserver_master_addrs=127.0.98.62:39275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:49:49.493722 17075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:49:49.494742 17075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 12:49:49.504388 17075 server_base.cc:1034] running on GCE node
W20260927 12:49:49.504340 17082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:49:49.504365 17083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:49:49.504343 17085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:49:49.505038 17075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:49:49.505707 17075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:49:49.506875 17075 hybrid_clock.cc:648] HybridClock initialized: now 1790513389506824 us; error 65 us; skew 500 ppm
Sep 27 12:49:49 dist-test-slave-lkn4 krb5kdc[15755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513389, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:49:49.509830 17075 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 12:49:49.510906 17075 webserver.cc:460] Webserver started at http://127.0.98.2:43429/ using document root <none> and password file <none>
I20260927 12:49:49.511554 17075 fs_manager.cc:359] Metadata directory not provided
I20260927 12:49:49.511624 17075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:49:49.515817 17075 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 12:49:49.518203 17092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:49:49.519644 17075 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:49:49.519773 17075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-1/wal
uuid: "eb3f8d8af27245ee986f70c3eba9975a"
format_stamp: "Formatted at 2026-09-27 12:49:07 on dist-test-slave-lkn4"
server_key: "f993f9f3b93280b8b7738b214cfc2938"
server_key_iv: "63b410bbfac9d7d130bb821ba47bd003"
server_key_version: "encryptionkey@0"
I20260927 12:49:49.520404 17075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:49:49.535480 17075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:49:49.538740 17075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:49:49.538929 17075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:49:49.539541 17075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:49:49.540683 17099 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260927 12:49:49.544422 17099 ts_tablet_manager.cc:1852] T 7bde087239f9498892c0c1bacad362a9 P eb3f8d8af27245ee986f70c3eba9975a: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260927 12:49:49.544811 17099 ts_tablet_manager.cc:1905] T 7bde087239f9498892c0c1bacad362a9 P eb3f8d8af27245ee986f70c3eba9975a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:49:49.548899 17099 ts_tablet_manager.cc:1918] T 7bde087239f9498892c0c1bacad362a9 P eb3f8d8af27245ee986f70c3eba9975a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:49:49.549001 17099 log.cc:1196] T 7bde087239f9498892c0c1bacad362a9 P eb3f8d8af27245ee986f70c3eba9975a: Deleting WAL directory at /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-1/wal/wals/7bde087239f9498892c0c1bacad362a9
I20260927 12:49:49.549342 17099 ts_tablet_manager.cc:1939] T 7bde087239f9498892c0c1bacad362a9 P eb3f8d8af27245ee986f70c3eba9975a: Deleting consensus metadata
I20260927 12:49:49.549625 17075 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260927 12:49:49.549686 17075 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.000s sys 0.001s
I20260927 12:49:49.549762 17075 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:49:49.549784 17075 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:49:49.559991 17075 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43565
I20260927 12:49:49.560077 17206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43565 every 8 connection(s)
I20260927 12:49:49.560967 17075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:49:49.564602 17209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:49.563216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39275 (local address 127.0.98.2:52269)
0927 12:49:49.563710 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:49.563830 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":116,"threads_started":1}
W20260927 12:49:49.565192 17207 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39275 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:39275: Client connection negotiation failed: client connection to 127.0.98.62:39275: connect: Connection refused (error 111)
I20260927 12:49:49.565804 17209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:49.565575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39275 (local address 127.0.98.2:49891)
0927 12:49:49.565696 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:49.565746 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 12:49:49.566540 17209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:49.566347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39275 (local address 127.0.98.2:56349)
0927 12:49:49.566445 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:49.566488 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 12:49:49.566730 17207 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 12:49:49.570820 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 17075
I20260927 12:49:49.571187 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:43905
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:39275
--webserver_port=37617
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:49:49.683547 17210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:49:49.683828 17210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:49:49.683873 17210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:49:49.688061 17210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:49:49.688131 17210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:49:49.688205 17210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:49:49.693984 17210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:43905
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=37617
--tserver_master_addrs=127.0.98.62:39275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:49:49.695211 17210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:49:49.696259 17210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:49:49.706094 17218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:49:49.706104 17216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:49:49.706072 17215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:49:49.706501 17210 server_base.cc:1034] running on GCE node
I20260927 12:49:49.707024 17210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:49:49.707774 17210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:49:49.708979 17210 hybrid_clock.cc:648] HybridClock initialized: now 1790513389708954 us; error 40 us; skew 500 ppm
Sep 27 12:49:49 dist-test-slave-lkn4 krb5kdc[15755](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513389, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:49:49.711862 17210 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 12:49:49.712904 17210 webserver.cc:460] Webserver started at http://127.0.98.3:37617/ using document root <none> and password file <none>
I20260927 12:49:49.713626 17210 fs_manager.cc:359] Metadata directory not provided
I20260927 12:49:49.713704 17210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:49:49.717929 17210 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.002s
I20260927 12:49:49.720394 17225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:49:49.721853 17210 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 12:49:49.721987 17210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-2/wal
uuid: "b81219f0f4124a45914289257fee36fe"
format_stamp: "Formatted at 2026-09-27 12:49:07 on dist-test-slave-lkn4"
server_key: "290953eaa5097417a0b56a4277a86de1"
server_key_iv: "6e470f406f9e23d155c00d2c35852494"
server_key_version: "encryptionkey@0"
I20260927 12:49:49.722675 17210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:49:49.739375 17210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:49:49.742807 17210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:49:49.742998 17210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:49:49.743603 17210 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:49:49.744459 17210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:49:49.744506 17210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:49:49.744561 17210 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:49:49.744588 17210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:49:49.753494 17210 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43905
I20260927 12:49:49.753517 17338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43905 every 8 connection(s)
I20260927 12:49:49.754534 17210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:49:49.757409 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 17210
I20260927 12:49:49.758827 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:49.756720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39275 (local address 127.0.98.3:33171)
0927 12:49:49.757280 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:49.757864 (+ 584us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":104,"threads_started":1}
W20260927 12:49:49.759441 17339 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39275 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:39275: Client connection negotiation failed: client connection to 127.0.98.62:39275: connect: Connection refused (error 111)
I20260927 12:49:49.760223 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:49.759963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39275 (local address 127.0.98.3:53291)
0927 12:49:49.760084 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:49.760134 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 12:49:49.761061 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:49.760781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39275 (local address 127.0.98.3:45545)
0927 12:49:49.760957 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:49.761005 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
W20260927 12:49:49.761257 17339 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 12:49:50.401796 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:50.401350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39275 (local address 127.0.98.1:46033)
0927 12:49:50.401622 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:50.401685 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":75,"threads_started":1}
I20260927 12:49:50.567806 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:50.567273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39275 (local address 127.0.98.2:39201)
0927 12:49:50.567559 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:50.567698 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":68,"threads_started":1}
I20260927 12:49:50.762334 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:50.761831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39275 (local address 127.0.98.3:36027)
0927 12:49:50.762138 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:50.762209 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":69,"threads_started":1}
I20260927 12:49:51.019239 17346 raft_consensus.cc:482] T 7bde087239f9498892c0c1bacad362a9 P 21c7b42976584d3caecaa0ddbea15cca [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:49:51.019476 17346 raft_consensus.cc:504] T 7bde087239f9498892c0c1bacad362a9 P 21c7b42976584d3caecaa0ddbea15cca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3f8d8af27245ee986f70c3eba9975a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43565 } } peers { permanent_uuid: "b81219f0f4124a45914289257fee36fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43905 } } peers { permanent_uuid: "21c7b42976584d3caecaa0ddbea15cca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41501 } }
I20260927 12:49:51.020576 17346 leader_election.cc:288] T 7bde087239f9498892c0c1bacad362a9 P 21c7b42976584d3caecaa0ddbea15cca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eb3f8d8af27245ee986f70c3eba9975a (127.0.98.2:43565), b81219f0f4124a45914289257fee36fe (127.0.98.3:43905)
Sep 27 12:49:51 dist-test-slave-lkn4 krb5kdc[15755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513389, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 27 12:49:51 dist-test-slave-lkn4 krb5kdc[15755](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513389, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260927 12:49:51.034154 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:51.020874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43565 (local address 127.0.98.1:44505)
0927 12:49:51.021304 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:51.021318 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:49:51.021438 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:49:51.022954 (+ 1516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:49:51.022961 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:49:51.023515 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:49:51.024088 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:49:51.024112 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:49:51.025374 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:49:51.025377 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:49:51.026390 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:49:51.026399 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:49:51.027139 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:49:51.027142 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:49:51.027369 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:49:51.028009 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:49:51.028025 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:49:51.029968 (+ 1943us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:49:51.031876 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:49:51.031879 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:49:51.031881 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:49:51.032106 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:49:51.032431 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:49:51.032434 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:49:51.032436 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:49:51.032559 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:49:51.033157 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:49:51.033163 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:49:51.033415 (+ 252us) client_negotiation.cc:769] Sending connection context
0927 12:49:51.033576 (+ 161us) client_negotiation.cc:240] Negotiation successful
0927 12:49:51.033915 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":264,"spinlock_wait_cycles":41856,"thread_start_us":68,"threads_started":1}
I20260927 12:49:51.034157 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:51.020970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44505 (local address 127.0.98.2:43565)
0927 12:49:51.021406 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0927 12:49:51.021424 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 12:49:51.021471 (+ 47us) server_negotiation.cc:407] Connection header received
0927 12:49:51.022284 (+ 813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:49:51.022291 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:49:51.022619 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:49:51.022895 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:49:51.024249 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:49:51.025243 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:49:51.026495 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:49:51.027028 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:49:51.027242 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:49:51.030151 (+ 2909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:49:51.030156 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:49:51.030167 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:49:51.030223 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:49:51.031772 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:49:51.032213 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:49:51.032218 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:49:51.032224 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:49:51.032335 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:49:51.032641 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:49:51.032645 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:49:51.032646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:49:51.033047 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:49:51.033168 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 12:49:51.033701 (+ 533us) server_negotiation.cc:299] Negotiation successful
0927 12:49:51.033944 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":65,"threads_started":1}
I20260927 12:49:51.034209 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:51.020874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43905 (local address 127.0.98.1:54665)
0927 12:49:51.021251 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:51.021270 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 12:49:51.021389 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:49:51.022958 (+ 1569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:49:51.022962 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:49:51.023495 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:49:51.024088 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:49:51.024105 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:49:51.025437 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:49:51.025441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:49:51.026392 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:49:51.026403 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:49:51.027135 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:49:51.027140 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:49:51.027372 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:49:51.028053 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:49:51.028066 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:49:51.029602 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:49:51.031804 (+ 2202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:49:51.031810 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:49:51.031825 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:49:51.032104 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:49:51.032465 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:49:51.032468 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:49:51.032470 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:49:51.032562 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:49:51.033159 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:49:51.033164 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:49:51.033412 (+ 248us) client_negotiation.cc:769] Sending connection context
0927 12:49:51.033577 (+ 165us) client_negotiation.cc:240] Negotiation successful
0927 12:49:51.033995 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":258,"spinlock_wait_cycles":16384,"thread_start_us":89,"threads_started":1}
I20260927 12:49:51.034418 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:51.021201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54665 (local address 127.0.98.3:43905)
0927 12:49:51.021596 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0927 12:49:51.021619 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 12:49:51.021650 (+ 31us) server_negotiation.cc:407] Connection header received
0927 12:49:51.022394 (+ 744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:49:51.022401 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:49:51.022690 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:49:51.022947 (+ 257us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:49:51.024256 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:49:51.025302 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:49:51.026511 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:49:51.027010 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:49:51.027235 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:49:51.029842 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:49:51.029848 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:49:51.029860 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:49:51.029891 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:49:51.031670 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:49:51.032212 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:49:51.032217 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:49:51.032222 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:49:51.032374 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:49:51.032647 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:49:51.032650 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:49:51.032651 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:49:51.033049 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:49:51.033168 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 12:49:51.033704 (+ 536us) server_negotiation.cc:299] Negotiation successful
0927 12:49:51.034277 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":99,"threads_started":1}
W20260927 12:49:51.036918 16961 leader_election.cc:341] T 7bde087239f9498892c0c1bacad362a9 P 21c7b42976584d3caecaa0ddbea15cca [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b81219f0f4124a45914289257fee36fe (127.0.98.3:43905): Not found: Tablet not found: 7bde087239f9498892c0c1bacad362a9
W20260927 12:49:51.037051 16959 leader_election.cc:341] T 7bde087239f9498892c0c1bacad362a9 P 21c7b42976584d3caecaa0ddbea15cca [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eb3f8d8af27245ee986f70c3eba9975a (127.0.98.2:43565): Not found: Tablet not found: 7bde087239f9498892c0c1bacad362a9
I20260927 12:49:51.037149 16959 leader_election.cc:302] T 7bde087239f9498892c0c1bacad362a9 P 21c7b42976584d3caecaa0ddbea15cca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 21c7b42976584d3caecaa0ddbea15cca; no voters: b81219f0f4124a45914289257fee36fe, eb3f8d8af27245ee986f70c3eba9975a
I20260927 12:49:51.037418 17346 raft_consensus.cc:2726] T 7bde087239f9498892c0c1bacad362a9 P 21c7b42976584d3caecaa0ddbea15cca [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 12:49:51.402962 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:51.402619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39275 (local address 127.0.98.1:57811)
0927 12:49:51.402788 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:51.402877 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 12:49:51.569011 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:51.568575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39275 (local address 127.0.98.2:37369)
0927 12:49:51.568840 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:51.568901 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":77,"threads_started":1}
I20260927 12:49:51.763612 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:51.763176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39275 (local address 127.0.98.3:41595)
0927 12:49:51.763447 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:51.763512 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":77,"threads_started":1}
I20260927 12:49:52.404248 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:52.403812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39275 (local address 127.0.98.1:40245)
0927 12:49:52.404074 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:52.404143 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":72,"threads_started":1}
I20260927 12:49:52.570248 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:52.569779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39275 (local address 127.0.98.2:59215)
0927 12:49:52.570080 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:52.570140 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":92,"threads_started":1}
I20260927 12:49:52.690596 17355 raft_consensus.cc:482] T 7bde087239f9498892c0c1bacad362a9 P 21c7b42976584d3caecaa0ddbea15cca [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:49:52.690816 17355 raft_consensus.cc:504] T 7bde087239f9498892c0c1bacad362a9 P 21c7b42976584d3caecaa0ddbea15cca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3f8d8af27245ee986f70c3eba9975a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43565 } } peers { permanent_uuid: "b81219f0f4124a45914289257fee36fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43905 } } peers { permanent_uuid: "21c7b42976584d3caecaa0ddbea15cca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41501 } }
I20260927 12:49:52.691337 17355 leader_election.cc:288] T 7bde087239f9498892c0c1bacad362a9 P 21c7b42976584d3caecaa0ddbea15cca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eb3f8d8af27245ee986f70c3eba9975a (127.0.98.2:43565), b81219f0f4124a45914289257fee36fe (127.0.98.3:43905)
W20260927 12:49:52.692237 16961 leader_election.cc:341] T 7bde087239f9498892c0c1bacad362a9 P 21c7b42976584d3caecaa0ddbea15cca [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b81219f0f4124a45914289257fee36fe (127.0.98.3:43905): Not found: Tablet not found: 7bde087239f9498892c0c1bacad362a9
W20260927 12:49:52.692399 16959 leader_election.cc:341] T 7bde087239f9498892c0c1bacad362a9 P 21c7b42976584d3caecaa0ddbea15cca [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eb3f8d8af27245ee986f70c3eba9975a (127.0.98.2:43565): Not found: Tablet not found: 7bde087239f9498892c0c1bacad362a9
I20260927 12:49:52.692497 16959 leader_election.cc:302] T 7bde087239f9498892c0c1bacad362a9 P 21c7b42976584d3caecaa0ddbea15cca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 21c7b42976584d3caecaa0ddbea15cca; no voters: b81219f0f4124a45914289257fee36fe, eb3f8d8af27245ee986f70c3eba9975a
I20260927 12:49:52.692648 17355 raft_consensus.cc:2726] T 7bde087239f9498892c0c1bacad362a9 P 21c7b42976584d3caecaa0ddbea15cca [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 12:49:52.764899 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:52.764455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39275 (local address 127.0.98.3:53227)
0927 12:49:52.764717 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:52.764783 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":82,"threads_started":1}
I20260927 12:49:53.405567 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:53.405077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39275 (local address 127.0.98.1:33579)
0927 12:49:53.405391 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:53.405466 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":87,"threads_started":1}
I20260927 12:49:53.571461 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:53.571048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39275 (local address 127.0.98.2:45127)
0927 12:49:53.571308 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:53.571369 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":75,"threads_started":1}
I20260927 12:49:53.766222 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:53.765735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39275 (local address 127.0.98.3:59467)
0927 12:49:53.766023 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:53.766118 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":76,"threads_started":1}
I20260927 12:49:54.406977 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:54.406517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39275 (local address 127.0.98.1:56459)
0927 12:49:54.406818 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:54.406879 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":76,"threads_started":1}
I20260927 12:49:54.572690 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:54.572238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39275 (local address 127.0.98.2:50361)
0927 12:49:54.572532 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:54.572592 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":101,"threads_started":1}
I20260927 12:49:54.767433 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:54.767002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39275 (local address 127.0.98.3:37237)
0927 12:49:54.767282 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:54.767338 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":87,"threads_started":1}
I20260927 12:49:55.408233 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:55.407802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39275 (local address 127.0.98.1:55847)
0927 12:49:55.408073 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:55.408134 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":73,"threads_started":1}
I20260927 12:49:55.574002 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:55.573544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39275 (local address 127.0.98.2:44111)
0927 12:49:55.573833 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:55.573903 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":80,"threads_started":1}
I20260927 12:49:55.719074 17365 raft_consensus.cc:482] T 7bde087239f9498892c0c1bacad362a9 P 21c7b42976584d3caecaa0ddbea15cca [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:49:55.719224 17365 raft_consensus.cc:504] T 7bde087239f9498892c0c1bacad362a9 P 21c7b42976584d3caecaa0ddbea15cca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3f8d8af27245ee986f70c3eba9975a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43565 } } peers { permanent_uuid: "b81219f0f4124a45914289257fee36fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43905 } } peers { permanent_uuid: "21c7b42976584d3caecaa0ddbea15cca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41501 } }
I20260927 12:49:55.719616 17365 leader_election.cc:288] T 7bde087239f9498892c0c1bacad362a9 P 21c7b42976584d3caecaa0ddbea15cca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eb3f8d8af27245ee986f70c3eba9975a (127.0.98.2:43565), b81219f0f4124a45914289257fee36fe (127.0.98.3:43905)
W20260927 12:49:55.720340 16961 leader_election.cc:341] T 7bde087239f9498892c0c1bacad362a9 P 21c7b42976584d3caecaa0ddbea15cca [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b81219f0f4124a45914289257fee36fe (127.0.98.3:43905): Not found: Tablet not found: 7bde087239f9498892c0c1bacad362a9
W20260927 12:49:55.720461 16959 leader_election.cc:341] T 7bde087239f9498892c0c1bacad362a9 P 21c7b42976584d3caecaa0ddbea15cca [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eb3f8d8af27245ee986f70c3eba9975a (127.0.98.2:43565): Not found: Tablet not found: 7bde087239f9498892c0c1bacad362a9
I20260927 12:49:55.720546 16959 leader_election.cc:302] T 7bde087239f9498892c0c1bacad362a9 P 21c7b42976584d3caecaa0ddbea15cca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 21c7b42976584d3caecaa0ddbea15cca; no voters: b81219f0f4124a45914289257fee36fe, eb3f8d8af27245ee986f70c3eba9975a
I20260927 12:49:55.720687 17365 raft_consensus.cc:2726] T 7bde087239f9498892c0c1bacad362a9 P 21c7b42976584d3caecaa0ddbea15cca [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 12:49:55.768546 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:55.768146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39275 (local address 127.0.98.3:59425)
0927 12:49:55.768393 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:55.768452 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":89,"threads_started":1}
I20260927 12:49:56.409466 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:56.408987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39275 (local address 127.0.98.1:37231)
0927 12:49:56.409299 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:56.409366 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":127,"threads_started":1}
I20260927 12:49:56.575403 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:56.574869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39275 (local address 127.0.98.2:38157)
0927 12:49:56.575225 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:56.575295 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":78,"threads_started":1}
I20260927 12:49:56.769723 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:56.769266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39275 (local address 127.0.98.3:46967)
0927 12:49:56.769574 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:56.769625 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":91,"threads_started":1}
I20260927 12:49:57.410741 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:57.410263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39275 (local address 127.0.98.1:49789)
0927 12:49:57.410515 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:57.410585 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":70,"threads_started":1}
I20260927 12:49:57.576750 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:57.576209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39275 (local address 127.0.98.2:60257)
0927 12:49:57.576517 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:57.576622 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":83,"threads_started":1}
I20260927 12:49:57.771093 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:57.770620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39275 (local address 127.0.98.3:42177)
0927 12:49:57.770926 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:57.770980 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":125,"threads_started":1}
I20260927 12:49:58.087668 17373 raft_consensus.cc:482] T 7bde087239f9498892c0c1bacad362a9 P 21c7b42976584d3caecaa0ddbea15cca [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:49:58.087807 17373 raft_consensus.cc:504] T 7bde087239f9498892c0c1bacad362a9 P 21c7b42976584d3caecaa0ddbea15cca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3f8d8af27245ee986f70c3eba9975a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43565 } } peers { permanent_uuid: "b81219f0f4124a45914289257fee36fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43905 } } peers { permanent_uuid: "21c7b42976584d3caecaa0ddbea15cca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41501 } }
I20260927 12:49:58.088239 17373 leader_election.cc:288] T 7bde087239f9498892c0c1bacad362a9 P 21c7b42976584d3caecaa0ddbea15cca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eb3f8d8af27245ee986f70c3eba9975a (127.0.98.2:43565), b81219f0f4124a45914289257fee36fe (127.0.98.3:43905)
W20260927 12:49:58.088981 16959 leader_election.cc:341] T 7bde087239f9498892c0c1bacad362a9 P 21c7b42976584d3caecaa0ddbea15cca [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eb3f8d8af27245ee986f70c3eba9975a (127.0.98.2:43565): Not found: Tablet not found: 7bde087239f9498892c0c1bacad362a9
W20260927 12:49:58.089082 16961 leader_election.cc:341] T 7bde087239f9498892c0c1bacad362a9 P 21c7b42976584d3caecaa0ddbea15cca [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b81219f0f4124a45914289257fee36fe (127.0.98.3:43905): Not found: Tablet not found: 7bde087239f9498892c0c1bacad362a9
I20260927 12:49:58.089267 16961 leader_election.cc:302] T 7bde087239f9498892c0c1bacad362a9 P 21c7b42976584d3caecaa0ddbea15cca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 21c7b42976584d3caecaa0ddbea15cca; no voters: b81219f0f4124a45914289257fee36fe, eb3f8d8af27245ee986f70c3eba9975a
I20260927 12:49:58.089445 17373 raft_consensus.cc:2726] T 7bde087239f9498892c0c1bacad362a9 P 21c7b42976584d3caecaa0ddbea15cca [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 12:49:58.411938 17374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:58.411426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39275 (local address 127.0.98.1:47659)
0927 12:49:58.411773 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:58.411836 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":81,"threads_started":1}
I20260927 12:49:58.578159 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:58.577634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39275 (local address 127.0.98.2:59737)
0927 12:49:58.577951 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:58.578018 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":89,"threads_started":1}
I20260927 12:49:58.772470 17376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:58.771963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39275 (local address 127.0.98.3:40683)
0927 12:49:58.772278 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:58.772351 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":99,"threads_started":1}
I20260927 12:49:59.413257 17377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:59.412822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39275 (local address 127.0.98.1:48809)
0927 12:49:59.413102 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:59.413162 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":72,"threads_started":1}
I20260927 12:49:59.579310 17378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:59.578903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39275 (local address 127.0.98.2:49405)
0927 12:49:59.579152 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:59.579210 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":76,"threads_started":1}
I20260927 12:49:59.773862 17379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:49:59.773432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39275 (local address 127.0.98.3:43377)
0927 12:49:59.773690 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0927 12:49:59.773760 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":83,"threads_started":1}
I20260927 12:50:00.414597 17380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:00.414141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39275 (local address 127.0.98.1:53497)
0927 12:50:00.414429 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:00.414495 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":74,"threads_started":1}
I20260927 12:50:00.580592 17381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:00.580165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39275 (local address 127.0.98.2:39955)
0927 12:50:00.580436 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:00.580496 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":77,"threads_started":1}
I20260927 12:50:00.775154 17382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:00.774692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39275 (local address 127.0.98.3:47383)
0927 12:50:00.774991 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:00.775056 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":92,"threads_started":1}
I20260927 12:50:01.415985 17383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:01.415370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39275 (local address 127.0.98.1:41941)
0927 12:50:01.415766 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:01.415858 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":93,"threads_started":1}
I20260927 12:50:01.582010 17384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:01.581448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39275 (local address 127.0.98.2:55879)
0927 12:50:01.581803 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:01.581887 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":103,"threads_started":1}
I20260927 12:50:01.776618 17385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:01.776140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39275 (local address 127.0.98.3:53357)
0927 12:50:01.776436 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:01.776511 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":80,"threads_started":1}
I20260927 12:50:02.417366 17386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:02.416936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39275 (local address 127.0.98.1:51073)
0927 12:50:02.417193 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:02.417264 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":72,"threads_started":1}
I20260927 12:50:02.583421 17387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:02.582959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39275 (local address 127.0.98.2:48203)
0927 12:50:02.583244 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:02.583321 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":101,"threads_started":1}
I20260927 12:50:02.777942 17388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:02.777465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39275 (local address 127.0.98.3:54901)
0927 12:50:02.777724 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:02.777842 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":74,"threads_started":1}
I20260927 12:50:03.418565 17389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:03.418128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39275 (local address 127.0.98.1:42949)
0927 12:50:03.418401 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:03.418457 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":79,"threads_started":1}
I20260927 12:50:03.584771 17390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:03.584281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39275 (local address 127.0.98.2:46149)
0927 12:50:03.584571 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:03.584657 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":74,"threads_started":1}
I20260927 12:50:03.779121 17391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:03.778680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39275 (local address 127.0.98.3:33443)
0927 12:50:03.778956 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:03.779013 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":107,"threads_started":1}
I20260927 12:50:04.295533 17392 raft_consensus.cc:482] T 7bde087239f9498892c0c1bacad362a9 P 21c7b42976584d3caecaa0ddbea15cca [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:50:04.295681 17392 raft_consensus.cc:504] T 7bde087239f9498892c0c1bacad362a9 P 21c7b42976584d3caecaa0ddbea15cca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb3f8d8af27245ee986f70c3eba9975a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43565 } } peers { permanent_uuid: "b81219f0f4124a45914289257fee36fe" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43905 } } peers { permanent_uuid: "21c7b42976584d3caecaa0ddbea15cca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41501 } }
I20260927 12:50:04.296078 17392 leader_election.cc:288] T 7bde087239f9498892c0c1bacad362a9 P 21c7b42976584d3caecaa0ddbea15cca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eb3f8d8af27245ee986f70c3eba9975a (127.0.98.2:43565), b81219f0f4124a45914289257fee36fe (127.0.98.3:43905)
W20260927 12:50:04.296870 16959 leader_election.cc:341] T 7bde087239f9498892c0c1bacad362a9 P 21c7b42976584d3caecaa0ddbea15cca [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eb3f8d8af27245ee986f70c3eba9975a (127.0.98.2:43565): Not found: Tablet not found: 7bde087239f9498892c0c1bacad362a9
W20260927 12:50:04.296994 16961 leader_election.cc:341] T 7bde087239f9498892c0c1bacad362a9 P 21c7b42976584d3caecaa0ddbea15cca [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b81219f0f4124a45914289257fee36fe (127.0.98.3:43905): Not found: Tablet not found: 7bde087239f9498892c0c1bacad362a9
I20260927 12:50:04.297068 16961 leader_election.cc:302] T 7bde087239f9498892c0c1bacad362a9 P 21c7b42976584d3caecaa0ddbea15cca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 21c7b42976584d3caecaa0ddbea15cca; no voters: b81219f0f4124a45914289257fee36fe, eb3f8d8af27245ee986f70c3eba9975a
I20260927 12:50:04.297245 17392 raft_consensus.cc:2726] T 7bde087239f9498892c0c1bacad362a9 P 21c7b42976584d3caecaa0ddbea15cca [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 12:50:04.420145 17393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:04.419609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39275 (local address 127.0.98.1:55195)
0927 12:50:04.419946 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:04.420015 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":79,"threads_started":1}
I20260927 12:50:04.586107 17394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:04.585682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39275 (local address 127.0.98.2:39385)
0927 12:50:04.585921 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:04.585970 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":76,"threads_started":1}
I20260927 12:50:04.762310 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 16902
I20260927 12:50:04.767442 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 17075
I20260927 12:50:04.772102 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 17210
2026-09-27T12:50:04Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59303 ms)
[----------] 30 tests from SecurityITest (243603 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:50:04 dist-test-slave-lkn4 krb5kdc[17397](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:50:04 dist-test-slave-lkn4 krb5kdc[17397](info): set up 2 sockets
Sep 27 12:50:04 dist-test-slave-lkn4 krb5kdc[17397](info): commencing operation
krb5kdc: starting...
W20260927 12:50:05.730425 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/krb5kdc/test-user.keytab.
Sep 27 12:50:05 dist-test-slave-lkn4 krb5kdc[17397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:50:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:50:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:50:05.853760 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35103
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35103
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:50:05.964983 17410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:50:05.965296 17410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:50:05.965353 17410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:50:05.969483 17410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:50:05.969555 17410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:50:05.969583 17410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:50:05.969602 17410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:50:05.969623 17410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:50:05.975092 17410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35103
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:50:05.976220 17410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:50:05.977257 17410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:50:05.985514 17415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:50:05.985520 17418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:50:05.985607 17410 server_base.cc:1034] running on GCE node
W20260927 12:50:05.985513 17416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:50:05.986342 17410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:50:05.987536 17410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:50:05.988776 17410 hybrid_clock.cc:648] HybridClock initialized: now 1790513405988762 us; error 63 us; skew 500 ppm
Sep 27 12:50:05 dist-test-slave-lkn4 krb5kdc[17397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513405, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:50:05.991636 17410 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:50:05.992576 17410 webserver.cc:460] Webserver started at http://127.0.98.62:42803/ using document root <none> and password file <none>
I20260927 12:50:05.993145 17410 fs_manager.cc:359] Metadata directory not provided
I20260927 12:50:05.993204 17410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:50:05.993373 17410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:50:05.995508 17410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/minicluster-data/master-0/data/instance:
uuid: "67a01b37e6ed4544bd5bed9370c2c146"
format_stamp: "Formatted at 2026-09-27 12:50:05 on dist-test-slave-lkn4"
server_key: "9295b97037818f61f1bc84dde20e1cf3"
server_key_iv: "d8b508e523e8d8730cf95ce8d2856014"
server_key_version: "encryptionkey@0"
I20260927 12:50:05.996032 17410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/minicluster-data/master-0/wal/instance:
uuid: "67a01b37e6ed4544bd5bed9370c2c146"
format_stamp: "Formatted at 2026-09-27 12:50:05 on dist-test-slave-lkn4"
server_key: "9295b97037818f61f1bc84dde20e1cf3"
server_key_iv: "d8b508e523e8d8730cf95ce8d2856014"
server_key_version: "encryptionkey@0"
I20260927 12:50:05.999976 17410 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260927 12:50:06.002542 17425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:06.003938 17410 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:50:06.004076 17410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/minicluster-data/master-0/wal
uuid: "67a01b37e6ed4544bd5bed9370c2c146"
format_stamp: "Formatted at 2026-09-27 12:50:05 on dist-test-slave-lkn4"
server_key: "9295b97037818f61f1bc84dde20e1cf3"
server_key_iv: "d8b508e523e8d8730cf95ce8d2856014"
server_key_version: "encryptionkey@0"
I20260927 12:50:06.004191 17410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:50:06.019618 17410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:50:06.022946 17410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:50:06.023134 17410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:50:06.029943 17410 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35103
I20260927 12:50:06.029986 17477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35103 every 8 connection(s)
I20260927 12:50:06.030979 17410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:50:06.033987 17478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:50:06.039924 17478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67a01b37e6ed4544bd5bed9370c2c146: Bootstrap starting.
I20260927 12:50:06.040354 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 17410
I20260927 12:50:06.040478 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/minicluster-data/master-0/wal/instance
I20260927 12:50:06.040781 392 external_mini_cluster.cc:1428] Setting key b8bf935a1daba54bdb96aef7c82436d9
I20260927 12:50:06.042886 17478 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67a01b37e6ed4544bd5bed9370c2c146: Neither blocks nor log segments found. Creating new log.
I20260927 12:50:06.043666 17478 log.cc:824] T 00000000000000000000000000000000 P 67a01b37e6ed4544bd5bed9370c2c146: Log is configured to *not* fsync() on all Append() calls
I20260927 12:50:06.045934 17478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67a01b37e6ed4544bd5bed9370c2c146: No bootstrap required, opened a new log
I20260927 12:50:06.049973 17478 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67a01b37e6ed4544bd5bed9370c2c146 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67a01b37e6ed4544bd5bed9370c2c146" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35103 } }
Sep 27 12:50:06 dist-test-slave-lkn4 krb5kdc[17397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:50:06.050264 17478 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67a01b37e6ed4544bd5bed9370c2c146 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:50:06.050347 17478 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67a01b37e6ed4544bd5bed9370c2c146 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67a01b37e6ed4544bd5bed9370c2c146, State: Initialized, Role: FOLLOWER
I20260927 12:50:06.050891 17478 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67a01b37e6ed4544bd5bed9370c2c146 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67a01b37e6ed4544bd5bed9370c2c146" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35103 } }
I20260927 12:50:06.051033 17478 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67a01b37e6ed4544bd5bed9370c2c146 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:50:06.051080 17478 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67a01b37e6ed4544bd5bed9370c2c146 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:50:06.051177 17478 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67a01b37e6ed4544bd5bed9370c2c146 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:50:06.052392 17478 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67a01b37e6ed4544bd5bed9370c2c146 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67a01b37e6ed4544bd5bed9370c2c146" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35103 } }
I20260927 12:50:06.052752 17478 leader_election.cc:302] T 00000000000000000000000000000000 P 67a01b37e6ed4544bd5bed9370c2c146 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67a01b37e6ed4544bd5bed9370c2c146; no voters:
I20260927 12:50:06.053066 17478 leader_election.cc:288] T 00000000000000000000000000000000 P 67a01b37e6ed4544bd5bed9370c2c146 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:50:06.053165 17483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67a01b37e6ed4544bd5bed9370c2c146 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:50:06.053385 17483 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67a01b37e6ed4544bd5bed9370c2c146 [term 1 LEADER]: Becoming Leader. State: Replica: 67a01b37e6ed4544bd5bed9370c2c146, State: Running, Role: LEADER
I20260927 12:50:06.053731 17483 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67a01b37e6ed4544bd5bed9370c2c146 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67a01b37e6ed4544bd5bed9370c2c146" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35103 } }
I20260927 12:50:06.054422 17478 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67a01b37e6ed4544bd5bed9370c2c146 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:50:06.055419 17484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67a01b37e6ed4544bd5bed9370c2c146 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67a01b37e6ed4544bd5bed9370c2c146" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67a01b37e6ed4544bd5bed9370c2c146" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35103 } } }
I20260927 12:50:06.055549 17484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67a01b37e6ed4544bd5bed9370c2c146 [sys.catalog]: This master's current role is: LEADER
I20260927 12:50:06.055859 17485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67a01b37e6ed4544bd5bed9370c2c146 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67a01b37e6ed4544bd5bed9370c2c146. Latest consensus state: current_term: 1 leader_uuid: "67a01b37e6ed4544bd5bed9370c2c146" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67a01b37e6ed4544bd5bed9370c2c146" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35103 } } }
I20260927 12:50:06.055915 17492 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:50:06.056012 17485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67a01b37e6ed4544bd5bed9370c2c146 [sys.catalog]: This master's current role is: LEADER
I20260927 12:50:06.056412 17481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:06.042323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45046 (local address 127.0.98.62:35103)
0927 12:50:06.042810 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:06.042826 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:06.042865 (+ 39us) server_negotiation.cc:407] Connection header received
0927 12:50:06.043614 (+ 749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:06.043635 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:06.044027 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:06.044341 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:06.045077 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:06.046542 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:06.047520 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:06.048241 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:06.048421 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:06.050914 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:06.050919 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:06.050934 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:50:06.050963 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:06.052950 (+ 1987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:06.053520 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:06.053526 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:06.053532 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:06.053671 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:06.053966 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:06.053970 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:06.053972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:06.054515 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:06.054736 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:06.055008 (+ 272us) server_negotiation.cc:299] Negotiation successful
0927 12:50:06.055354 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":129,"threads_started":1}
I20260927 12:50:06.059262 17492 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:50:06.065320 17492 catalog_manager.cc:1347] Generated new cluster ID: 0c13c9ee905548a992cadfc1bca60d31
I20260927 12:50:06.065403 17492 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:50:06.088572 17492 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:50:06.090065 17492 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:50:06.102685 17492 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67a01b37e6ed4544bd5bed9370c2c146: Generated new TSK 0
I20260927 12:50:06.103574 17492 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 12:50:06.133468 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35103
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:50:06.245350 17504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:50:06.245625 17504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:50:06.245676 17504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:50:06.249668 17504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:50:06.249725 17504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:50:06.249784 17504 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:50:06.255290 17504 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17504
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:50:06.256417 17504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:50:06.257462 17504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 12:50:06.267203 17504 server_base.cc:1034] running on GCE node
W20260927 12:50:06.267150 17509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:50:06.267186 17510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:50:06.267150 17512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:50:06.267823 17504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:50:06.268426 17504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:50:06.269586 17504 hybrid_clock.cc:648] HybridClock initialized: now 1790513406269556 us; error 51 us; skew 500 ppm
Sep 27 12:50:06 dist-test-slave-lkn4 krb5kdc[17397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513406, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:50:06.272495 17504 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 12:50:06.273455 17504 webserver.cc:460] Webserver started at http://127.0.98.1:38533/ using document root <none> and password file <none>
I20260927 12:50:06.274000 17504 fs_manager.cc:359] Metadata directory not provided
I20260927 12:50:06.274097 17504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:50:06.274307 17504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:50:06.276338 17504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/minicluster-data/ts-0/data/instance:
uuid: "8f0b7771659c48f79b60993fbfa5fd89"
format_stamp: "Formatted at 2026-09-27 12:50:06 on dist-test-slave-lkn4"
server_key: "3c65d3f138d81828b6238e720f944c19"
server_key_iv: "818ca9abcb8adb706dfc5554ae9bb373"
server_key_version: "encryptionkey@0"
I20260927 12:50:06.276872 17504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8f0b7771659c48f79b60993fbfa5fd89"
format_stamp: "Formatted at 2026-09-27 12:50:06 on dist-test-slave-lkn4"
server_key: "3c65d3f138d81828b6238e720f944c19"
server_key_iv: "818ca9abcb8adb706dfc5554ae9bb373"
server_key_version: "encryptionkey@0"
I20260927 12:50:06.280810 17504 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260927 12:50:06.283424 17519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:06.284807 17504 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 12:50:06.284935 17504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/minicluster-data/ts-0/wal
uuid: "8f0b7771659c48f79b60993fbfa5fd89"
format_stamp: "Formatted at 2026-09-27 12:50:06 on dist-test-slave-lkn4"
server_key: "3c65d3f138d81828b6238e720f944c19"
server_key_iv: "818ca9abcb8adb706dfc5554ae9bb373"
server_key_version: "encryptionkey@0"
I20260927 12:50:06.285058 17504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:50:06.303452 17504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:50:06.306460 17504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:50:06.306636 17504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:50:06.307220 17504 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:50:06.308193 17504 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:50:06.308241 17504 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:06.308293 17504 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:50:06.308323 17504 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:06.317109 17504 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43341
I20260927 12:50:06.317119 17632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43341 every 8 connection(s)
I20260927 12:50:06.318032 17504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:50:06.319665 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 17504
I20260927 12:50:06.319839 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:50:06.320178 392 external_mini_cluster.cc:1428] Setting key 164ff9db12f232029c09a45825be6633
Sep 27 12:50:06 dist-test-slave-lkn4 krb5kdc[17397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513406, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 12:50:06.332723 17481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:06.319926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43553 (local address 127.0.98.62:35103)
0927 12:50:06.320107 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:06.320112 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:06.321268 (+ 1156us) server_negotiation.cc:407] Connection header received
0927 12:50:06.322368 (+ 1100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:06.322370 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:06.322414 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:06.322481 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:06.323995 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:06.324521 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:06.326131 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:06.326365 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:06.326444 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:06.329260 (+ 2816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:06.329263 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:06.329265 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:50:06.329291 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:06.330682 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:06.331266 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:06.331269 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:06.331271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:06.331358 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:06.331637 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:06.331639 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:06.331640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:06.331816 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:06.331896 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:06.332448 (+ 552us) server_negotiation.cc:299] Negotiation successful
0927 12:50:06.332569 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 12:50:06.333940 17635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:06.320386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35103 (local address 127.0.98.1:43553)
0927 12:50:06.321117 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:06.321162 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 12:50:06.322144 (+ 982us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:50:06.322763 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:50:06.322774 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:50:06.323255 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:50:06.323843 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:06.323858 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:06.324651 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:06.324654 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:06.325970 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:06.325982 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:06.326511 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:06.326522 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:06.326736 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:06.327336 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:50:06.327359 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:50:06.329078 (+ 1719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:50:06.330828 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:50:06.330834 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:50:06.330847 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:50:06.331166 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:50:06.331428 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:50:06.331430 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:50:06.331432 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:50:06.331574 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:50:06.331898 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:50:06.331904 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:50:06.332210 (+ 306us) client_negotiation.cc:769] Sending connection context
0927 12:50:06.332388 (+ 178us) client_negotiation.cc:240] Negotiation successful
0927 12:50:06.332853 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":142,"threads_started":1}
I20260927 12:50:06.335280 17633 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35103
I20260927 12:50:06.335552 17633 heartbeater.cc:464] Registering TS with master...
I20260927 12:50:06.336067 17633 heartbeater.cc:511] Master 127.0.98.62:35103 requested a full tablet report, sending...
I20260927 12:50:06.337854 17442 ts_manager.cc:194] Registered new tserver with Master: 8f0b7771659c48f79b60993fbfa5fd89 (127.0.98.1:43341)
I20260927 12:50:06.339058 17442 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43553
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 12:50:06.349274 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35103
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:50:06.458962 17638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:50:06.459259 17638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:50:06.459306 17638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:50:06.463415 17638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:50:06.463491 17638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:50:06.463577 17638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:50:06.469023 17638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:50:06.470245 17638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:50:06.471297 17638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:50:06.480799 17643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:50:06.480811 17644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:50:06.480877 17638 server_base.cc:1034] running on GCE node
W20260927 12:50:06.480826 17646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:50:06.481449 17638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:50:06.482101 17638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:50:06.483289 17638 hybrid_clock.cc:648] HybridClock initialized: now 1790513406483268 us; error 31 us; skew 500 ppm
Sep 27 12:50:06 dist-test-slave-lkn4 krb5kdc[17397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513406, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:50:06.486132 17638 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 12:50:06.487136 17638 webserver.cc:460] Webserver started at http://127.0.98.2:42945/ using document root <none> and password file <none>
I20260927 12:50:06.487722 17638 fs_manager.cc:359] Metadata directory not provided
I20260927 12:50:06.487792 17638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:50:06.487989 17638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:50:06.490259 17638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/minicluster-data/ts-1/data/instance:
uuid: "96a6b658477c4d159319aa0b76b9f1af"
format_stamp: "Formatted at 2026-09-27 12:50:06 on dist-test-slave-lkn4"
server_key: "a7b35714e0c40a8cf59ef162a154681c"
server_key_iv: "279e22996d97bdf4ccfd5b6aba3368aa"
server_key_version: "encryptionkey@0"
I20260927 12:50:06.490809 17638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/minicluster-data/ts-1/wal/instance:
uuid: "96a6b658477c4d159319aa0b76b9f1af"
format_stamp: "Formatted at 2026-09-27 12:50:06 on dist-test-slave-lkn4"
server_key: "a7b35714e0c40a8cf59ef162a154681c"
server_key_iv: "279e22996d97bdf4ccfd5b6aba3368aa"
server_key_version: "encryptionkey@0"
I20260927 12:50:06.494719 17638 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 12:50:06.497291 17653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:06.498806 17638 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 12:50:06.498929 17638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/minicluster-data/ts-1/wal
uuid: "96a6b658477c4d159319aa0b76b9f1af"
format_stamp: "Formatted at 2026-09-27 12:50:06 on dist-test-slave-lkn4"
server_key: "a7b35714e0c40a8cf59ef162a154681c"
server_key_iv: "279e22996d97bdf4ccfd5b6aba3368aa"
server_key_version: "encryptionkey@0"
I20260927 12:50:06.499033 17638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:50:06.512174 17638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:50:06.515774 17638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:50:06.516012 17638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:50:06.516637 17638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:50:06.517505 17638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:50:06.517554 17638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:06.517609 17638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:50:06.517638 17638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:06.526285 17638 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33229
I20260927 12:50:06.526311 17766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33229 every 8 connection(s)
I20260927 12:50:06.527217 17638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:50:06.535794 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 17638
I20260927 12:50:06.535936 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:50:06.536309 392 external_mini_cluster.cc:1428] Setting key 8d997d3ecaee20a6dfb4db488b7e4236
Sep 27 12:50:06 dist-test-slave-lkn4 krb5kdc[17397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513406, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:50:06.541922 17481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:06.529174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52883 (local address 127.0.98.62:35103)
0927 12:50:06.529342 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:06.529346 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:06.530251 (+ 905us) server_negotiation.cc:407] Connection header received
0927 12:50:06.531147 (+ 896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:06.531150 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:06.531205 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:06.531283 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:06.532528 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:06.533066 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:06.534324 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:06.534557 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:06.534754 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:06.537746 (+ 2992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:06.537751 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:06.537754 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:50:06.537782 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:06.539354 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:06.540073 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:06.540076 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:06.540077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:06.540162 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:06.540569 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:06.540572 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:06.540574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:06.540846 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:06.540935 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:06.541617 (+ 682us) server_negotiation.cc:299] Negotiation successful
0927 12:50:06.541794 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 12:50:06.543154 17769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:06.529579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35103 (local address 127.0.98.2:52883)
0927 12:50:06.530092 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:06.530141 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 12:50:06.530949 (+ 808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:50:06.531459 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:50:06.531466 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:50:06.531858 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:50:06.532378 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:06.532392 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:06.533172 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:06.533174 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:06.534192 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:06.534202 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:06.534811 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:06.534815 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:06.534969 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:06.535620 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:50:06.535642 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:50:06.537457 (+ 1815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:50:06.539505 (+ 2048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:50:06.539513 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:50:06.539533 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:50:06.539968 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:50:06.540249 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:50:06.540252 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:50:06.540254 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:50:06.540452 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:50:06.540963 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:50:06.540970 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:50:06.541340 (+ 370us) client_negotiation.cc:769] Sending connection context
0927 12:50:06.541568 (+ 228us) client_negotiation.cc:240] Negotiation successful
0927 12:50:06.541950 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":118,"threads_started":1}
I20260927 12:50:06.544380 17767 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35103
I20260927 12:50:06.544674 17767 heartbeater.cc:464] Registering TS with master...
I20260927 12:50:06.545343 17767 heartbeater.cc:511] Master 127.0.98.62:35103 requested a full tablet report, sending...
I20260927 12:50:06.546691 17442 ts_manager.cc:194] Registered new tserver with Master: 96a6b658477c4d159319aa0b76b9f1af (127.0.98.2:33229)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 12:50:06.547465 17442 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52883
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 12:50:06.565891 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35103
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:50:06.678646 17772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:50:06.678931 17772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:50:06.678975 17772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:50:06.683105 17772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:50:06.683179 17772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:50:06.683244 17772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:50:06.690701 17772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35103
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17772
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:50:06.691817 17772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:50:06.692852 17772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:50:06.702531 17778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:50:06.702558 17780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:50:06.702592 17777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:50:06.702746 17772 server_base.cc:1034] running on GCE node
I20260927 12:50:06.703284 17772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:50:06.703948 17772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:50:06.705139 17772 hybrid_clock.cc:648] HybridClock initialized: now 1790513406705125 us; error 26 us; skew 500 ppm
Sep 27 12:50:06 dist-test-slave-lkn4 krb5kdc[17397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513406, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:50:06.708068 17772 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 12:50:06.709120 17772 webserver.cc:460] Webserver started at http://127.0.98.3:34095/ using document root <none> and password file <none>
I20260927 12:50:06.709702 17772 fs_manager.cc:359] Metadata directory not provided
I20260927 12:50:06.709769 17772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:50:06.709964 17772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:50:06.712090 17772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/minicluster-data/ts-2/data/instance:
uuid: "8614feaa3b2b4f858da70ad87355c5d1"
format_stamp: "Formatted at 2026-09-27 12:50:06 on dist-test-slave-lkn4"
server_key: "64bc535c1a663a73dbb347e02b134afc"
server_key_iv: "19d3f4a029c10d5d50d6470cb81b8070"
server_key_version: "encryptionkey@0"
I20260927 12:50:06.712684 17772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8614feaa3b2b4f858da70ad87355c5d1"
format_stamp: "Formatted at 2026-09-27 12:50:06 on dist-test-slave-lkn4"
server_key: "64bc535c1a663a73dbb347e02b134afc"
server_key_iv: "19d3f4a029c10d5d50d6470cb81b8070"
server_key_version: "encryptionkey@0"
I20260927 12:50:06.716686 17772 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 12:50:06.719470 17787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:06.720825 17772 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 12:50:06.720948 17772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/minicluster-data/ts-2/wal
uuid: "8614feaa3b2b4f858da70ad87355c5d1"
format_stamp: "Formatted at 2026-09-27 12:50:06 on dist-test-slave-lkn4"
server_key: "64bc535c1a663a73dbb347e02b134afc"
server_key_iv: "19d3f4a029c10d5d50d6470cb81b8070"
server_key_version: "encryptionkey@0"
I20260927 12:50:06.721055 17772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:50:06.734184 17772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:50:06.737371 17772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:50:06.737560 17772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:50:06.738250 17772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:50:06.739084 17772 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:50:06.739131 17772 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:06.739185 17772 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:50:06.739214 17772 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:06.747855 17772 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32889
I20260927 12:50:06.747886 17900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32889 every 8 connection(s)
I20260927 12:50:06.748739 17772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:50:06.752594 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 17772
I20260927 12:50:06.752760 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:50:06.753165 392 external_mini_cluster.cc:1428] Setting key 4e967976304c1059f1996dca013960d6
Sep 27 12:50:06 dist-test-slave-lkn4 krb5kdc[17397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513406, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:50:06.763099 17481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:06.750531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47589 (local address 127.0.98.62:35103)
0927 12:50:06.750766 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:06.750771 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:06.751927 (+ 1156us) server_negotiation.cc:407] Connection header received
0927 12:50:06.753029 (+ 1102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:06.753032 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:06.753086 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:06.753187 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:06.754645 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:06.755394 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:06.756786 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:06.757002 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:06.757090 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:06.759590 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:06.759593 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:06.759595 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:50:06.759621 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:06.761101 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:06.761567 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:06.761570 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:06.761571 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:06.761648 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:06.762000 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:06.762002 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:06.762003 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:06.762247 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:06.762322 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:06.762855 (+ 533us) server_negotiation.cc:299] Negotiation successful
0927 12:50:06.762974 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 12:50:06.763959 17903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:06.750967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35103 (local address 127.0.98.3:47589)
0927 12:50:06.751739 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:06.751788 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 12:50:06.752764 (+ 976us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:50:06.753387 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:50:06.753398 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:50:06.753822 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:50:06.754490 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:06.754513 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:06.755532 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:06.755535 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:06.756667 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:06.756677 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:06.757143 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:06.757150 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:06.757312 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:06.757911 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:50:06.757928 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:50:06.759423 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:50:06.761211 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:50:06.761216 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:50:06.761229 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:50:06.761482 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:50:06.761772 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:50:06.761774 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:50:06.761775 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:50:06.761921 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:50:06.762335 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:50:06.762340 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:50:06.762590 (+ 250us) client_negotiation.cc:769] Sending connection context
0927 12:50:06.762762 (+ 172us) client_negotiation.cc:240] Negotiation successful
0927 12:50:06.763035 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":115,"threads_started":1}
I20260927 12:50:06.765204 17901 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35103
I20260927 12:50:06.765455 17901 heartbeater.cc:464] Registering TS with master...
I20260927 12:50:06.766124 17901 heartbeater.cc:511] Master 127.0.98.62:35103 requested a full tablet report, sending...
I20260927 12:50:06.767355 17442 ts_manager.cc:194] Registered new tserver with Master: 8614feaa3b2b4f858da70ad87355c5d1 (127.0.98.3:32889)
I20260927 12:50:06.767920 17442 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47589
I20260927 12:50:06.779826 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 12:50:06.789379 17481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:06.781843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45052 (local address 127.0.98.62:35103)
0927 12:50:06.781970 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:06.781973 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:06.782083 (+ 110us) server_negotiation.cc:407] Connection header received
0927 12:50:06.782224 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:06.782226 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:06.782270 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:06.782343 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:06.783185 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:06.783735 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:06.784603 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:06.784830 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:06.784912 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:06.786347 (+ 1435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:06.786350 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:06.786352 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:50:06.786379 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:06.787632 (+ 1253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:06.788075 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:06.788078 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:06.788080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:06.788160 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:06.788443 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:06.788446 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:06.788447 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:06.788678 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:06.788841 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:06.789122 (+ 281us) server_negotiation.cc:299] Negotiation successful
0927 12:50:06.789252 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 12:50:06.792554 17442 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45052:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:50:06.795715 17442 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:50:06.810401 17915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:06.804659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43341 (local address 127.0.0.1:48106)
0927 12:50:06.805453 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:06.805466 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:50:06.805554 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:50:06.806278 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:50:06.806284 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:50:06.806305 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:50:06.806464 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:06.806471 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:06.807489 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:06.807492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:06.808893 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:06.808903 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:06.809804 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:06.809808 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:06.809909 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:06.809956 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 12:50:06.810109 (+ 153us) client_negotiation.cc:240] Negotiation successful
0927 12:50:06.810244 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":731,"mutex_wait_us":49,"thread_start_us":103,"threads_started":1}
I20260927 12:50:06.810415 17916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:06.805309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48106 (local address 127.0.98.1:43341)
0927 12:50:06.805781 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:06.805796 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:06.805820 (+ 24us) server_negotiation.cc:407] Connection header received
0927 12:50:06.805896 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:06.805902 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:06.806129 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:06.806277 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:50:06.806589 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:06.807371 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:06.809005 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:06.809680 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:06.809781 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:06.809928 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:06.810151 (+ 223us) server_negotiation.cc:299] Negotiation successful
0927 12:50:06.810254 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":93,"threads_started":1}
I20260927 12:50:06.810532 17914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:06.805215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32889 (local address 127.0.0.1:58866)
0927 12:50:06.805829 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:06.805842 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:50:06.805942 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:50:06.806461 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:50:06.806463 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:50:06.806478 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:50:06.806557 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:06.806562 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:06.807920 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:06.807923 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:06.809121 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:06.809131 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:06.810205 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:06.810209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:06.810287 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:06.810308 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 12:50:06.810373 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 12:50:06.810431 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"mutex_wait_us":1,"thread_start_us":95,"threads_started":1}
I20260927 12:50:06.810854 17917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:06.805466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58866 (local address 127.0.98.3:32889)
0927 12:50:06.805848 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:06.805860 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:06.805910 (+ 50us) server_negotiation.cc:407] Connection header received
0927 12:50:06.806086 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:06.806092 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:06.806298 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:06.806435 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:50:06.806639 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:06.807779 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:06.809252 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:06.809899 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:06.809993 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:06.810150 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:06.810617 (+ 467us) server_negotiation.cc:299] Negotiation successful
0927 12:50:06.810706 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":111,"threads_started":1}
I20260927 12:50:06.810751 17912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:06.804501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33229 (local address 127.0.0.1:39006)
0927 12:50:06.804945 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:06.804993 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 12:50:06.805114 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:50:06.806637 (+ 1523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:50:06.806640 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:50:06.806657 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:50:06.806742 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:06.806748 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:06.808190 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:06.808193 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:06.809313 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:06.809321 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:06.810341 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:06.810345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:06.810441 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:06.810466 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 12:50:06.810592 (+ 126us) client_negotiation.cc:240] Negotiation successful
0927 12:50:06.810660 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":2,"thread_start_us":85,"threads_started":1}
I20260927 12:50:06.811415 17913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:06.804667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39006 (local address 127.0.98.2:33229)
0927 12:50:06.806228 (+ 1561us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:06.806244 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:06.806264 (+ 20us) server_negotiation.cc:407] Connection header received
0927 12:50:06.806336 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:06.806341 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:06.806499 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:06.806886 (+ 387us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:50:06.807038 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:06.808060 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:06.809418 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:06.810212 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:06.810833 (+ 621us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:06.811124 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:06.811175 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 12:50:06.811259 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1456,"thread_start_us":96,"threads_started":1}
I20260927 12:50:06.813889 17701 tablet_service.cc:1467] Processing CreateTablet for tablet 1969b79f80fe401eba49339604b692e9 (DEFAULT_TABLE table=test-table [id=60878e6dfaf0469bb2e2b689e08c97c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:50:06.813910 17835 tablet_service.cc:1467] Processing CreateTablet for tablet 1969b79f80fe401eba49339604b692e9 (DEFAULT_TABLE table=test-table [id=60878e6dfaf0469bb2e2b689e08c97c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:50:06.813912 17567 tablet_service.cc:1467] Processing CreateTablet for tablet 1969b79f80fe401eba49339604b692e9 (DEFAULT_TABLE table=test-table [id=60878e6dfaf0469bb2e2b689e08c97c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:50:06.814919 17701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1969b79f80fe401eba49339604b692e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:50:06.814945 17567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1969b79f80fe401eba49339604b692e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:50:06.814945 17835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1969b79f80fe401eba49339604b692e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:50:06.821250 17918 tablet_bootstrap.cc:492] T 1969b79f80fe401eba49339604b692e9 P 96a6b658477c4d159319aa0b76b9f1af: Bootstrap starting.
I20260927 12:50:06.822633 17920 tablet_bootstrap.cc:492] T 1969b79f80fe401eba49339604b692e9 P 8614feaa3b2b4f858da70ad87355c5d1: Bootstrap starting.
I20260927 12:50:06.822710 17919 tablet_bootstrap.cc:492] T 1969b79f80fe401eba49339604b692e9 P 8f0b7771659c48f79b60993fbfa5fd89: Bootstrap starting.
I20260927 12:50:06.823778 17918 tablet_bootstrap.cc:654] T 1969b79f80fe401eba49339604b692e9 P 96a6b658477c4d159319aa0b76b9f1af: Neither blocks nor log segments found. Creating new log.
I20260927 12:50:06.824714 17918 log.cc:824] T 1969b79f80fe401eba49339604b692e9 P 96a6b658477c4d159319aa0b76b9f1af: Log is configured to *not* fsync() on all Append() calls
I20260927 12:50:06.824831 17919 tablet_bootstrap.cc:654] T 1969b79f80fe401eba49339604b692e9 P 8f0b7771659c48f79b60993fbfa5fd89: Neither blocks nor log segments found. Creating new log.
I20260927 12:50:06.824930 17920 tablet_bootstrap.cc:654] T 1969b79f80fe401eba49339604b692e9 P 8614feaa3b2b4f858da70ad87355c5d1: Neither blocks nor log segments found. Creating new log.
I20260927 12:50:06.826109 17920 log.cc:824] T 1969b79f80fe401eba49339604b692e9 P 8614feaa3b2b4f858da70ad87355c5d1: Log is configured to *not* fsync() on all Append() calls
I20260927 12:50:06.826109 17919 log.cc:824] T 1969b79f80fe401eba49339604b692e9 P 8f0b7771659c48f79b60993fbfa5fd89: Log is configured to *not* fsync() on all Append() calls
I20260927 12:50:06.827296 17918 tablet_bootstrap.cc:492] T 1969b79f80fe401eba49339604b692e9 P 96a6b658477c4d159319aa0b76b9f1af: No bootstrap required, opened a new log
I20260927 12:50:06.827489 17918 ts_tablet_manager.cc:1397] T 1969b79f80fe401eba49339604b692e9 P 96a6b658477c4d159319aa0b76b9f1af: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 12:50:06.828078 17919 tablet_bootstrap.cc:492] T 1969b79f80fe401eba49339604b692e9 P 8f0b7771659c48f79b60993fbfa5fd89: No bootstrap required, opened a new log
I20260927 12:50:06.828080 17920 tablet_bootstrap.cc:492] T 1969b79f80fe401eba49339604b692e9 P 8614feaa3b2b4f858da70ad87355c5d1: No bootstrap required, opened a new log
I20260927 12:50:06.828276 17920 ts_tablet_manager.cc:1397] T 1969b79f80fe401eba49339604b692e9 P 8614feaa3b2b4f858da70ad87355c5d1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 12:50:06.828276 17919 ts_tablet_manager.cc:1397] T 1969b79f80fe401eba49339604b692e9 P 8f0b7771659c48f79b60993fbfa5fd89: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 12:50:06.830492 17918 raft_consensus.cc:348] T 1969b79f80fe401eba49339604b692e9 P 96a6b658477c4d159319aa0b76b9f1af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96a6b658477c4d159319aa0b76b9f1af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33229 } } peers { permanent_uuid: "8f0b7771659c48f79b60993fbfa5fd89" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43341 } } peers { permanent_uuid: "8614feaa3b2b4f858da70ad87355c5d1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32889 } }
I20260927 12:50:06.830747 17918 raft_consensus.cc:374] T 1969b79f80fe401eba49339604b692e9 P 96a6b658477c4d159319aa0b76b9f1af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:50:06.830819 17918 raft_consensus.cc:729] T 1969b79f80fe401eba49339604b692e9 P 96a6b658477c4d159319aa0b76b9f1af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96a6b658477c4d159319aa0b76b9f1af, State: Initialized, Role: FOLLOWER
I20260927 12:50:06.831278 17918 consensus_queue.cc:260] T 1969b79f80fe401eba49339604b692e9 P 96a6b658477c4d159319aa0b76b9f1af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96a6b658477c4d159319aa0b76b9f1af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33229 } } peers { permanent_uuid: "8f0b7771659c48f79b60993fbfa5fd89" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43341 } } peers { permanent_uuid: "8614feaa3b2b4f858da70ad87355c5d1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32889 } }
I20260927 12:50:06.831995 17767 heartbeater.cc:503] Master 127.0.98.62:35103 was elected leader, sending a full tablet report...
I20260927 12:50:06.832294 17918 ts_tablet_manager.cc:1428] T 1969b79f80fe401eba49339604b692e9 P 96a6b658477c4d159319aa0b76b9f1af: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 12:50:06.832165 17919 raft_consensus.cc:348] T 1969b79f80fe401eba49339604b692e9 P 8f0b7771659c48f79b60993fbfa5fd89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96a6b658477c4d159319aa0b76b9f1af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33229 } } peers { permanent_uuid: "8f0b7771659c48f79b60993fbfa5fd89" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43341 } } peers { permanent_uuid: "8614feaa3b2b4f858da70ad87355c5d1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32889 } }
I20260927 12:50:06.832643 17919 raft_consensus.cc:374] T 1969b79f80fe401eba49339604b692e9 P 8f0b7771659c48f79b60993fbfa5fd89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:50:06.832738 17919 raft_consensus.cc:729] T 1969b79f80fe401eba49339604b692e9 P 8f0b7771659c48f79b60993fbfa5fd89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f0b7771659c48f79b60993fbfa5fd89, State: Initialized, Role: FOLLOWER
I20260927 12:50:06.833215 17919 consensus_queue.cc:260] T 1969b79f80fe401eba49339604b692e9 P 8f0b7771659c48f79b60993fbfa5fd89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96a6b658477c4d159319aa0b76b9f1af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33229 } } peers { permanent_uuid: "8f0b7771659c48f79b60993fbfa5fd89" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43341 } } peers { permanent_uuid: "8614feaa3b2b4f858da70ad87355c5d1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32889 } }
I20260927 12:50:06.833235 17920 raft_consensus.cc:348] T 1969b79f80fe401eba49339604b692e9 P 8614feaa3b2b4f858da70ad87355c5d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96a6b658477c4d159319aa0b76b9f1af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33229 } } peers { permanent_uuid: "8f0b7771659c48f79b60993fbfa5fd89" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43341 } } peers { permanent_uuid: "8614feaa3b2b4f858da70ad87355c5d1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32889 } }
I20260927 12:50:06.833565 17920 raft_consensus.cc:374] T 1969b79f80fe401eba49339604b692e9 P 8614feaa3b2b4f858da70ad87355c5d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:50:06.833654 17920 raft_consensus.cc:729] T 1969b79f80fe401eba49339604b692e9 P 8614feaa3b2b4f858da70ad87355c5d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8614feaa3b2b4f858da70ad87355c5d1, State: Initialized, Role: FOLLOWER
I20260927 12:50:06.834141 17633 heartbeater.cc:503] Master 127.0.98.62:35103 was elected leader, sending a full tablet report...
I20260927 12:50:06.834187 17920 consensus_queue.cc:260] T 1969b79f80fe401eba49339604b692e9 P 8614feaa3b2b4f858da70ad87355c5d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96a6b658477c4d159319aa0b76b9f1af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33229 } } peers { permanent_uuid: "8f0b7771659c48f79b60993fbfa5fd89" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43341 } } peers { permanent_uuid: "8614feaa3b2b4f858da70ad87355c5d1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32889 } }
I20260927 12:50:06.834595 17919 ts_tablet_manager.cc:1428] T 1969b79f80fe401eba49339604b692e9 P 8f0b7771659c48f79b60993fbfa5fd89: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 12:50:06.835167 17920 ts_tablet_manager.cc:1428] T 1969b79f80fe401eba49339604b692e9 P 8614feaa3b2b4f858da70ad87355c5d1: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260927 12:50:06.835491 17901 heartbeater.cc:503] Master 127.0.98.62:35103 was elected leader, sending a full tablet report...
I20260927 12:50:06.927915 17924 raft_consensus.cc:482] T 1969b79f80fe401eba49339604b692e9 P 96a6b658477c4d159319aa0b76b9f1af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:50:06.928138 17924 raft_consensus.cc:504] T 1969b79f80fe401eba49339604b692e9 P 96a6b658477c4d159319aa0b76b9f1af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96a6b658477c4d159319aa0b76b9f1af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33229 } } peers { permanent_uuid: "8f0b7771659c48f79b60993fbfa5fd89" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43341 } } peers { permanent_uuid: "8614feaa3b2b4f858da70ad87355c5d1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32889 } }
I20260927 12:50:06.929391 17924 leader_election.cc:288] T 1969b79f80fe401eba49339604b692e9 P 96a6b658477c4d159319aa0b76b9f1af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f0b7771659c48f79b60993fbfa5fd89 (127.0.98.1:43341), 8614feaa3b2b4f858da70ad87355c5d1 (127.0.98.3:32889)
I20260927 12:50:06.933003 17917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:06.929750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47975 (local address 127.0.98.3:32889)
0927 12:50:06.929884 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:06.929889 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:06.929905 (+ 16us) server_negotiation.cc:407] Connection header received
0927 12:50:06.930104 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:06.930107 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:06.930154 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:06.930221 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:50:06.930462 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:06.931023 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:06.932047 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:06.932464 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:06.932556 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:06.932619 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:06.932818 (+ 199us) server_negotiation.cc:299] Negotiation successful
0927 12:50:06.932884 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 12:50:06.933003 17769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:06.929647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32889 (local address 127.0.98.2:47975)
0927 12:50:06.929842 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:06.929857 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:50:06.929953 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:50:06.930244 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:50:06.930247 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:50:06.930264 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:50:06.930368 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:06.930373 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:06.931133 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:06.931137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:06.931928 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:06.931937 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:06.932579 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:06.932583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:06.932693 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:06.932717 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 12:50:06.932781 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 12:50:06.932860 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":6}
I20260927 12:50:06.933097 17916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:06.929822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38343 (local address 127.0.98.1:43341)
0927 12:50:06.929956 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:06.929972 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:06.930182 (+ 210us) server_negotiation.cc:407] Connection header received
0927 12:50:06.930305 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:06.930308 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:06.930349 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:06.930408 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:50:06.930667 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:06.931143 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:06.932167 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:06.932563 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:06.932651 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:06.932717 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:06.932902 (+ 185us) server_negotiation.cc:299] Negotiation successful
0927 12:50:06.932976 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 12:50:06.933141 17927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:06.929661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43341 (local address 127.0.98.2:38343)
0927 12:50:06.930095 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:06.930107 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:50:06.930212 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:50:06.930429 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:50:06.930432 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:50:06.930446 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:50:06.930557 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:06.930562 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:06.931244 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:06.931247 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:06.932032 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:06.932048 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:06.932673 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:06.932676 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:06.932794 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:06.932813 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 12:50:06.932876 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 12:50:06.932945 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"mutex_wait_us":18,"thread_start_us":154,"threads_started":1}
I20260927 12:50:06.933813 17587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1969b79f80fe401eba49339604b692e9" candidate_uuid: "96a6b658477c4d159319aa0b76b9f1af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f0b7771659c48f79b60993fbfa5fd89" is_pre_election: true
I20260927 12:50:06.933813 17855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1969b79f80fe401eba49339604b692e9" candidate_uuid: "96a6b658477c4d159319aa0b76b9f1af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8614feaa3b2b4f858da70ad87355c5d1" is_pre_election: true
I20260927 12:50:06.934204 17587 raft_consensus.cc:2454] T 1969b79f80fe401eba49339604b692e9 P 8f0b7771659c48f79b60993fbfa5fd89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96a6b658477c4d159319aa0b76b9f1af in term 0.
I20260927 12:50:06.934204 17855 raft_consensus.cc:2454] T 1969b79f80fe401eba49339604b692e9 P 8614feaa3b2b4f858da70ad87355c5d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96a6b658477c4d159319aa0b76b9f1af in term 0.
I20260927 12:50:06.934716 17657 leader_election.cc:302] T 1969b79f80fe401eba49339604b692e9 P 96a6b658477c4d159319aa0b76b9f1af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8614feaa3b2b4f858da70ad87355c5d1, 96a6b658477c4d159319aa0b76b9f1af; no voters:
I20260927 12:50:06.934980 17924 raft_consensus.cc:2781] T 1969b79f80fe401eba49339604b692e9 P 96a6b658477c4d159319aa0b76b9f1af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:50:06.935061 17924 raft_consensus.cc:482] T 1969b79f80fe401eba49339604b692e9 P 96a6b658477c4d159319aa0b76b9f1af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:50:06.935109 17924 raft_consensus.cc:3037] T 1969b79f80fe401eba49339604b692e9 P 96a6b658477c4d159319aa0b76b9f1af [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:50:06.936143 17924 raft_consensus.cc:504] T 1969b79f80fe401eba49339604b692e9 P 96a6b658477c4d159319aa0b76b9f1af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96a6b658477c4d159319aa0b76b9f1af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33229 } } peers { permanent_uuid: "8f0b7771659c48f79b60993fbfa5fd89" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43341 } } peers { permanent_uuid: "8614feaa3b2b4f858da70ad87355c5d1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32889 } }
I20260927 12:50:06.936493 17924 leader_election.cc:288] T 1969b79f80fe401eba49339604b692e9 P 96a6b658477c4d159319aa0b76b9f1af [CANDIDATE]: Term 1 election: Requested vote from peers 8f0b7771659c48f79b60993fbfa5fd89 (127.0.98.1:43341), 8614feaa3b2b4f858da70ad87355c5d1 (127.0.98.3:32889)
I20260927 12:50:06.936879 17855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1969b79f80fe401eba49339604b692e9" candidate_uuid: "96a6b658477c4d159319aa0b76b9f1af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8614feaa3b2b4f858da70ad87355c5d1"
I20260927 12:50:06.937039 17587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1969b79f80fe401eba49339604b692e9" candidate_uuid: "96a6b658477c4d159319aa0b76b9f1af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f0b7771659c48f79b60993fbfa5fd89"
I20260927 12:50:06.937083 17855 raft_consensus.cc:3037] T 1969b79f80fe401eba49339604b692e9 P 8614feaa3b2b4f858da70ad87355c5d1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:50:06.937171 17587 raft_consensus.cc:3037] T 1969b79f80fe401eba49339604b692e9 P 8f0b7771659c48f79b60993fbfa5fd89 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:50:06.938221 17587 raft_consensus.cc:2454] T 1969b79f80fe401eba49339604b692e9 P 8f0b7771659c48f79b60993fbfa5fd89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96a6b658477c4d159319aa0b76b9f1af in term 1.
I20260927 12:50:06.938233 17855 raft_consensus.cc:2454] T 1969b79f80fe401eba49339604b692e9 P 8614feaa3b2b4f858da70ad87355c5d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96a6b658477c4d159319aa0b76b9f1af in term 1.
I20260927 12:50:06.938529 17656 leader_election.cc:302] T 1969b79f80fe401eba49339604b692e9 P 96a6b658477c4d159319aa0b76b9f1af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f0b7771659c48f79b60993fbfa5fd89, 96a6b658477c4d159319aa0b76b9f1af; no voters:
I20260927 12:50:06.938694 17924 raft_consensus.cc:2781] T 1969b79f80fe401eba49339604b692e9 P 96a6b658477c4d159319aa0b76b9f1af [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:50:06.938930 17924 raft_consensus.cc:686] T 1969b79f80fe401eba49339604b692e9 P 96a6b658477c4d159319aa0b76b9f1af [term 1 LEADER]: Becoming Leader. State: Replica: 96a6b658477c4d159319aa0b76b9f1af, State: Running, Role: LEADER
I20260927 12:50:06.939304 17924 consensus_queue.cc:237] T 1969b79f80fe401eba49339604b692e9 P 96a6b658477c4d159319aa0b76b9f1af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96a6b658477c4d159319aa0b76b9f1af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33229 } } peers { permanent_uuid: "8f0b7771659c48f79b60993fbfa5fd89" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43341 } } peers { permanent_uuid: "8614feaa3b2b4f858da70ad87355c5d1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32889 } }
I20260927 12:50:06.942502 17435 catalog_manager.cc:5495] T 1969b79f80fe401eba49339604b692e9 P 96a6b658477c4d159319aa0b76b9f1af reported cstate change: term changed from 0 to 1, leader changed from <none> to 96a6b658477c4d159319aa0b76b9f1af (127.0.98.2). New cstate: current_term: 1 leader_uuid: "96a6b658477c4d159319aa0b76b9f1af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96a6b658477c4d159319aa0b76b9f1af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33229 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f0b7771659c48f79b60993fbfa5fd89" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8614feaa3b2b4f858da70ad87355c5d1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32889 } health_report { overall_health: UNKNOWN } } }
I20260927 12:50:06.964242 17913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:06.960617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39012 (local address 127.0.98.2:33229)
0927 12:50:06.960757 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:06.960761 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:06.960794 (+ 33us) server_negotiation.cc:407] Connection header received
0927 12:50:06.960968 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:06.960971 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:06.961026 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:06.961125 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:50:06.961372 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:06.961871 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:06.962785 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:06.962998 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:06.963084 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:06.963377 (+ 293us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:50:06.963826 (+ 449us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:50:06.963937 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:06.963998 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 12:50:06.964061 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 12:50:06.971309 17855 raft_consensus.cc:1261] T 1969b79f80fe401eba49339604b692e9 P 8614feaa3b2b4f858da70ad87355c5d1 [term 1 FOLLOWER]: Refusing update from remote peer 96a6b658477c4d159319aa0b76b9f1af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:50:06.971316 17587 raft_consensus.cc:1261] T 1969b79f80fe401eba49339604b692e9 P 8f0b7771659c48f79b60993fbfa5fd89 [term 1 FOLLOWER]: Refusing update from remote peer 96a6b658477c4d159319aa0b76b9f1af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:50:06.972067 17928 consensus_queue.cc:1035] T 1969b79f80fe401eba49339604b692e9 P 96a6b658477c4d159319aa0b76b9f1af [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f0b7771659c48f79b60993fbfa5fd89" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:50:06.972265 17924 consensus_queue.cc:1035] T 1969b79f80fe401eba49339604b692e9 P 96a6b658477c4d159319aa0b76b9f1af [LEADER]: Connected to new peer: Peer: permanent_uuid: "8614feaa3b2b4f858da70ad87355c5d1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:50:06.989982 17440 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45052:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:50:06.990252 17440 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45052:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:50:06.992784 17440 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 67a01b37e6ed4544bd5bed9370c2c146: Sending DeleteTablet for 3 replicas of tablet 1969b79f80fe401eba49339604b692e9
I20260927 12:50:06.993708 17567 tablet_service.cc:1514] Processing DeleteTablet for tablet 1969b79f80fe401eba49339604b692e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:50:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48106
I20260927 12:50:06.993708 17835 tablet_service.cc:1514] Processing DeleteTablet for tablet 1969b79f80fe401eba49339604b692e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:50:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58866
I20260927 12:50:06.993907 17701 tablet_service.cc:1514] Processing DeleteTablet for tablet 1969b79f80fe401eba49339604b692e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:50:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39006
I20260927 12:50:06.994448 17942 tablet_replica.cc:330] T 1969b79f80fe401eba49339604b692e9 P 96a6b658477c4d159319aa0b76b9f1af: stopping tablet replica
I20260927 12:50:06.994802 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 17504
I20260927 12:50:06.994731 17941 tablet_replica.cc:330] T 1969b79f80fe401eba49339604b692e9 P 8614feaa3b2b4f858da70ad87355c5d1: stopping tablet replica
I20260927 12:50:06.996448 17942 raft_consensus.cc:2229] T 1969b79f80fe401eba49339604b692e9 P 96a6b658477c4d159319aa0b76b9f1af [term 1 LEADER]: Raft consensus shutting down.
I20260927 12:50:06.996876 17942 raft_consensus.cc:2258] T 1969b79f80fe401eba49339604b692e9 P 96a6b658477c4d159319aa0b76b9f1af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:50:06.997326 17941 raft_consensus.cc:2229] T 1969b79f80fe401eba49339604b692e9 P 8614feaa3b2b4f858da70ad87355c5d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:50:06.997685 17941 raft_consensus.cc:2258] T 1969b79f80fe401eba49339604b692e9 P 8614feaa3b2b4f858da70ad87355c5d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:50:06.999352 17941 ts_tablet_manager.cc:1905] T 1969b79f80fe401eba49339604b692e9 P 8614feaa3b2b4f858da70ad87355c5d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:50:06.999351 17942 ts_tablet_manager.cc:1905] T 1969b79f80fe401eba49339604b692e9 P 96a6b658477c4d159319aa0b76b9f1af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:50:07.002351 17942 ts_tablet_manager.cc:1918] T 1969b79f80fe401eba49339604b692e9 P 96a6b658477c4d159319aa0b76b9f1af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260927 12:50:07.002432 17428 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43341 (error 108)
I20260927 12:50:07.002513 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 17638
I20260927 12:50:07.006631 17941 ts_tablet_manager.cc:1918] T 1969b79f80fe401eba49339604b692e9 P 8614feaa3b2b4f858da70ad87355c5d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:50:07.006906 17941 log.cc:1196] T 1969b79f80fe401eba49339604b692e9 P 8614feaa3b2b4f858da70ad87355c5d1: Deleting WAL directory at /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790513161160093-392-0/minicluster-data/ts-2/wal/wals/1969b79f80fe401eba49339604b692e9
I20260927 12:50:07.007402 17941 ts_tablet_manager.cc:1939] T 1969b79f80fe401eba49339604b692e9 P 8614feaa3b2b4f858da70ad87355c5d1: Deleting consensus metadata
I20260927 12:50:07.007424 17915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:07.007072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43341 (local address 127.0.0.1:48114)
0927 12:50:07.007229 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:07.007320 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260927 12:50:07.008738 17428 catalog_manager.cc:4584] TS 8f0b7771659c48f79b60993fbfa5fd89 (127.0.98.1:43341): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1969b79f80fe401eba49339604b692e9: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43341: connect: Connection refused (error 111)
I20260927 12:50:07.009125 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 17772
I20260927 12:50:07.009310 17429 catalog_manager.cc:4841] TS 8614feaa3b2b4f858da70ad87355c5d1 (127.0.98.3:32889): tablet 1969b79f80fe401eba49339604b692e9 (table test-table [id=60878e6dfaf0469bb2e2b689e08c97c6]) successfully deleted
I20260927 12:50:07.011301 17915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:07.010347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33229 (local address 127.0.0.1:39020)
0927 12:50:07.011147 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:07.011209 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:33229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":708}
I20260927 12:50:07.016870 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 17410
2026-09-27T12:50:07Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2247 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-27T12:50:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:50:07Z Disabled control of system clock
I20260927 12:50:07.050186 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790513161160093-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790513161160093-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790513161160093-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43127
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43127
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 12:50:07.163419 17947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:50:07.163779 17947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:50:07.163816 17947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:50:07.168161 17947 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:50:07.168239 17947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:50:07.168254 17947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:50:07.168269 17947 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:50:07.168283 17947 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:50:07.173358 17947 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790513161160093-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790513161160093-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43127
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790513161160093-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:50:07.174590 17947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:50:07.176043 17947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:50:07.184430 17952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:50:07.184448 17953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:50:07.184651 17947 server_base.cc:1034] running on GCE node
W20260927 12:50:07.184446 17955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:50:07.185335 17947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:50:07.186511 17947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:50:07.187695 17947 hybrid_clock.cc:648] HybridClock initialized: now 1790513407187666 us; error 43 us; skew 500 ppm
I20260927 12:50:07.189674 17947 webserver.cc:460] Webserver started at http://127.0.98.62:46869/ using document root <none> and password file <none>
I20260927 12:50:07.190332 17947 fs_manager.cc:359] Metadata directory not provided
I20260927 12:50:07.190407 17947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:50:07.190613 17947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:50:07.192692 17947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790513161160093-392-0/minicluster-data/master-0/data/instance:
uuid: "0e42f9e22631430b8cbf3ed3a72ec64b"
format_stamp: "Formatted at 2026-09-27 12:50:07 on dist-test-slave-lkn4"
server_key: "00f60f3c52c9897d0fe3cf9ab8335b6e"
server_key_iv: "41bba67dfe236291203905cbfc9c5427"
server_key_version: "encryptionkey@0"
I20260927 12:50:07.193239 17947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790513161160093-392-0/minicluster-data/master-0/wal/instance:
uuid: "0e42f9e22631430b8cbf3ed3a72ec64b"
format_stamp: "Formatted at 2026-09-27 12:50:07 on dist-test-slave-lkn4"
server_key: "00f60f3c52c9897d0fe3cf9ab8335b6e"
server_key_iv: "41bba67dfe236291203905cbfc9c5427"
server_key_version: "encryptionkey@0"
I20260927 12:50:07.197038 17947 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 12:50:07.199779 17961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:07.201081 17947 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 12:50:07.201238 17947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790513161160093-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790513161160093-392-0/minicluster-data/master-0/wal
uuid: "0e42f9e22631430b8cbf3ed3a72ec64b"
format_stamp: "Formatted at 2026-09-27 12:50:07 on dist-test-slave-lkn4"
server_key: "00f60f3c52c9897d0fe3cf9ab8335b6e"
server_key_iv: "41bba67dfe236291203905cbfc9c5427"
server_key_version: "encryptionkey@0"
I20260927 12:50:07.201350 17947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790513161160093-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790513161160093-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790513161160093-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:50:07.219835 17947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:50:07.220623 17947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:50:07.220816 17947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:50:07.227887 17947 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43127
I20260927 12:50:07.227893 18013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43127 every 8 connection(s)
I20260927 12:50:07.228839 17947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:50:07.232234 18014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:50:07.237025 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 17947
I20260927 12:50:07.237164 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790513161160093-392-0/minicluster-data/master-0/wal/instance
I20260927 12:50:07.237428 392 external_mini_cluster.cc:1428] Setting key 2adc251678e3a35725c9e5b092197144
I20260927 12:50:07.239414 18014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e42f9e22631430b8cbf3ed3a72ec64b: Bootstrap starting.
I20260927 12:50:07.241869 18014 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e42f9e22631430b8cbf3ed3a72ec64b: Neither blocks nor log segments found. Creating new log.
I20260927 12:50:07.242686 18014 log.cc:824] T 00000000000000000000000000000000 P 0e42f9e22631430b8cbf3ed3a72ec64b: Log is configured to *not* fsync() on all Append() calls
I20260927 12:50:07.245044 18014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e42f9e22631430b8cbf3ed3a72ec64b: No bootstrap required, opened a new log
I20260927 12:50:07.248108 18017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:07.239126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43782 (local address 127.0.98.62:43127)
0927 12:50:07.239949 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:07.239965 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:07.240006 (+ 41us) server_negotiation.cc:407] Connection header received
0927 12:50:07.240737 (+ 731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:07.240757 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:07.241246 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:07.241593 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:07.242133 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:07.243250 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:07.244565 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:07.245070 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:07.245276 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:07.246169 (+ 893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:07.246174 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:07.246188 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:50:07.246591 (+ 403us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:07.246646 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:50:07.246667 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:07.246819 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:07.246997 (+ 178us) server_negotiation.cc:299] Negotiation successful
0927 12:50:07.247276 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":158,"threads_started":1}
I20260927 12:50:07.248996 18014 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e42f9e22631430b8cbf3ed3a72ec64b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e42f9e22631430b8cbf3ed3a72ec64b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43127 } }
I20260927 12:50:07.249277 18014 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e42f9e22631430b8cbf3ed3a72ec64b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:50:07.249337 18014 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e42f9e22631430b8cbf3ed3a72ec64b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e42f9e22631430b8cbf3ed3a72ec64b, State: Initialized, Role: FOLLOWER
I20260927 12:50:07.249794 18014 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e42f9e22631430b8cbf3ed3a72ec64b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e42f9e22631430b8cbf3ed3a72ec64b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43127 } }
I20260927 12:50:07.249943 18014 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e42f9e22631430b8cbf3ed3a72ec64b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:50:07.250005 18014 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e42f9e22631430b8cbf3ed3a72ec64b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:50:07.250119 18014 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e42f9e22631430b8cbf3ed3a72ec64b [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:50:07.251021 18014 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e42f9e22631430b8cbf3ed3a72ec64b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e42f9e22631430b8cbf3ed3a72ec64b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43127 } }
I20260927 12:50:07.251354 18014 leader_election.cc:302] T 00000000000000000000000000000000 P 0e42f9e22631430b8cbf3ed3a72ec64b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e42f9e22631430b8cbf3ed3a72ec64b; no voters:
I20260927 12:50:07.251614 18014 leader_election.cc:288] T 00000000000000000000000000000000 P 0e42f9e22631430b8cbf3ed3a72ec64b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:50:07.251734 18019 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e42f9e22631430b8cbf3ed3a72ec64b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:50:07.251945 18019 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e42f9e22631430b8cbf3ed3a72ec64b [term 1 LEADER]: Becoming Leader. State: Replica: 0e42f9e22631430b8cbf3ed3a72ec64b, State: Running, Role: LEADER
I20260927 12:50:07.252388 18019 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e42f9e22631430b8cbf3ed3a72ec64b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e42f9e22631430b8cbf3ed3a72ec64b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43127 } }
I20260927 12:50:07.252897 18014 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e42f9e22631430b8cbf3ed3a72ec64b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:50:07.253959 18021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e42f9e22631430b8cbf3ed3a72ec64b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e42f9e22631430b8cbf3ed3a72ec64b. Latest consensus state: current_term: 1 leader_uuid: "0e42f9e22631430b8cbf3ed3a72ec64b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e42f9e22631430b8cbf3ed3a72ec64b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43127 } } }
I20260927 12:50:07.254133 18021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e42f9e22631430b8cbf3ed3a72ec64b [sys.catalog]: This master's current role is: LEADER
I20260927 12:50:07.254190 18020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e42f9e22631430b8cbf3ed3a72ec64b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e42f9e22631430b8cbf3ed3a72ec64b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e42f9e22631430b8cbf3ed3a72ec64b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43127 } } }
I20260927 12:50:07.254292 18020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e42f9e22631430b8cbf3ed3a72ec64b [sys.catalog]: This master's current role is: LEADER
W20260927 12:50:07.257563 18034 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0e42f9e22631430b8cbf3ed3a72ec64b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 12:50:07.257642 18034 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 12:50:07.257707 18028 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:50:07.258432 18028 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:50:07.263474 18028 catalog_manager.cc:1347] Generated new cluster ID: 0abbd2db4b594ea987aa45674fabba0c
I20260927 12:50:07.263543 18028 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:50:07.278532 18028 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:50:07.279867 18028 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:50:07.292510 18028 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e42f9e22631430b8cbf3ed3a72ec64b: Generated new TSK 0
I20260927 12:50:07.293476 18028 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:50:07.297664 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790513161160093-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790513161160093-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790513161160093-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43127
--builtin_ntp_servers=127.0.98.20:45005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 12:50:07.409399 18038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:50:07.409696 18038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:50:07.409739 18038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:50:07.413941 18038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:50:07.414016 18038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:50:07.414096 18038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:50:07.419080 18038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790513161160093-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790513161160093-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790513161160093-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:50:07.420239 18038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:50:07.421694 18038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:50:07.431708 18046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:50:07.431784 18043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:50:07.431723 18038 server_base.cc:1034] running on GCE node
W20260927 12:50:07.431749 18044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:50:07.432390 18038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:50:07.433109 18038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:50:07.434311 18038 hybrid_clock.cc:648] HybridClock initialized: now 1790513407434283 us; error 44 us; skew 500 ppm
I20260927 12:50:07.436329 18038 webserver.cc:460] Webserver started at http://127.0.98.1:42739/ using document root <none> and password file <none>
I20260927 12:50:07.436931 18038 fs_manager.cc:359] Metadata directory not provided
I20260927 12:50:07.437000 18038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:50:07.437187 18038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:50:07.439381 18038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790513161160093-392-0/minicluster-data/ts-0/data/instance:
uuid: "53bebd9622f64e88bbc3d59a308b630b"
format_stamp: "Formatted at 2026-09-27 12:50:07 on dist-test-slave-lkn4"
server_key: "706291d49e0980404fb5f87fc27a34bd"
server_key_iv: "444962616e7d59d82a37c2702b882b20"
server_key_version: "encryptionkey@0"
I20260927 12:50:07.439925 18038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790513161160093-392-0/minicluster-data/ts-0/wal/instance:
uuid: "53bebd9622f64e88bbc3d59a308b630b"
format_stamp: "Formatted at 2026-09-27 12:50:07 on dist-test-slave-lkn4"
server_key: "706291d49e0980404fb5f87fc27a34bd"
server_key_iv: "444962616e7d59d82a37c2702b882b20"
server_key_version: "encryptionkey@0"
I20260927 12:50:07.443873 18038 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260927 12:50:07.446655 18052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:07.448000 18038 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:50:07.448129 18038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790513161160093-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790513161160093-392-0/minicluster-data/ts-0/wal
uuid: "53bebd9622f64e88bbc3d59a308b630b"
format_stamp: "Formatted at 2026-09-27 12:50:07 on dist-test-slave-lkn4"
server_key: "706291d49e0980404fb5f87fc27a34bd"
server_key_iv: "444962616e7d59d82a37c2702b882b20"
server_key_version: "encryptionkey@0"
I20260927 12:50:07.448239 18038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790513161160093-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790513161160093-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790513161160093-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:50:07.460919 18038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:50:07.461737 18038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:50:07.461951 18038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:50:07.462646 18038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:50:07.463663 18038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:50:07.463713 18038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:07.463778 18038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:50:07.463804 18038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:07.472612 18038 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43515
I20260927 12:50:07.472635 18165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43515 every 8 connection(s)
I20260927 12:50:07.473562 18038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:50:07.482776 18017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:07.475426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48165 (local address 127.0.98.62:43127)
0927 12:50:07.475586 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:07.475589 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:07.476666 (+ 1077us) server_negotiation.cc:407] Connection header received
0927 12:50:07.478085 (+ 1419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:07.478088 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:07.478145 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:07.478215 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:07.479122 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:07.479660 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:07.480800 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:07.481031 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:07.481124 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:07.482296 (+ 1172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:07.482299 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:07.482301 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:50:07.482315 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:07.482334 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:50:07.482342 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:07.482397 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:07.482546 (+ 149us) server_negotiation.cc:299] Negotiation successful
0927 12:50:07.482614 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 12:50:07.483814 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 18038
I20260927 12:50:07.483839 18168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:07.475848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43127 (local address 127.0.98.1:48165)
0927 12:50:07.476512 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:07.476553 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 12:50:07.477852 (+ 1299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:50:07.478397 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:50:07.478403 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:50:07.478492 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:50:07.478986 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:07.478999 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:07.479778 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:07.479781 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:07.480679 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:07.480688 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:07.481190 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:07.481195 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:07.481359 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:07.481975 (+ 616us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 12:50:07.481977 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:50:07.482023 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 12:50:07.482131 (+ 108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:50:07.482408 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:50:07.482418 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:50:07.482424 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 12:50:07.482538 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 12:50:07.482850 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":108,"threads_started":1}
I20260927 12:50:07.483934 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790513161160093-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:50:07.484272 392 external_mini_cluster.cc:1428] Setting key 5a48bbfeb423aa6a659fd255e8501e97
I20260927 12:50:07.485311 18166 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43127
I20260927 12:50:07.485585 18166 heartbeater.cc:464] Registering TS with master...
I20260927 12:50:07.486219 18166 heartbeater.cc:511] Master 127.0.98.62:43127 requested a full tablet report, sending...
I20260927 12:50:07.486580 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790513161160093-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790513161160093-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790513161160093-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43127
--builtin_ntp_servers=127.0.98.20:45005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 12:50:07.487887 17978 ts_manager.cc:194] Registered new tserver with Master: 53bebd9622f64e88bbc3d59a308b630b (127.0.98.1:43515)
I20260927 12:50:07.489122 17978 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:48165
W20260927 12:50:07.602375 18169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:50:07.602669 18169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:50:07.602706 18169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:50:07.606964 18169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:50:07.607033 18169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:50:07.607091 18169 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:50:07.612061 18169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790513161160093-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790513161160093-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790513161160093-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:50:07.613191 18169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:50:07.614609 18169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:50:07.624553 18175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:50:07.624570 18177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:50:07.624553 18174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:50:07.624894 18169 server_base.cc:1034] running on GCE node
I20260927 12:50:07.625335 18169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:50:07.626010 18169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:50:07.627203 18169 hybrid_clock.cc:648] HybridClock initialized: now 1790513407627173 us; error 48 us; skew 500 ppm
I20260927 12:50:07.629097 18169 webserver.cc:460] Webserver started at http://127.0.98.2:33349/ using document root <none> and password file <none>
I20260927 12:50:07.629647 18169 fs_manager.cc:359] Metadata directory not provided
I20260927 12:50:07.629711 18169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:50:07.629920 18169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:50:07.632059 18169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790513161160093-392-0/minicluster-data/ts-1/data/instance:
uuid: "05b7b16ecb644f58882ca57898d09c43"
format_stamp: "Formatted at 2026-09-27 12:50:07 on dist-test-slave-lkn4"
server_key: "231cd258225ed42642bb7c6642638180"
server_key_iv: "93d46da41de98d4363c63538e5f8c321"
server_key_version: "encryptionkey@0"
I20260927 12:50:07.632635 18169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790513161160093-392-0/minicluster-data/ts-1/wal/instance:
uuid: "05b7b16ecb644f58882ca57898d09c43"
format_stamp: "Formatted at 2026-09-27 12:50:07 on dist-test-slave-lkn4"
server_key: "231cd258225ed42642bb7c6642638180"
server_key_iv: "93d46da41de98d4363c63538e5f8c321"
server_key_version: "encryptionkey@0"
I20260927 12:50:07.636699 18169 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 12:50:07.639667 18183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:07.641265 18169 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 12:50:07.641412 18169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790513161160093-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790513161160093-392-0/minicluster-data/ts-1/wal
uuid: "05b7b16ecb644f58882ca57898d09c43"
format_stamp: "Formatted at 2026-09-27 12:50:07 on dist-test-slave-lkn4"
server_key: "231cd258225ed42642bb7c6642638180"
server_key_iv: "93d46da41de98d4363c63538e5f8c321"
server_key_version: "encryptionkey@0"
I20260927 12:50:07.641528 18169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790513161160093-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790513161160093-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790513161160093-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:50:07.661890 18169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:50:07.662684 18169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:50:07.662858 18169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:50:07.663416 18169 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:50:07.664438 18169 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:50:07.664489 18169 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:07.664531 18169 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:50:07.664546 18169 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:07.673439 18169 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38015
I20260927 12:50:07.673461 18296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38015 every 8 connection(s)
I20260927 12:50:07.674408 18169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:50:07.683569 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 18169
I20260927 12:50:07.683707 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790513161160093-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:50:07.684047 392 external_mini_cluster.cc:1428] Setting key 0936f8720874fe0c6891564c6849abaa
I20260927 12:50:07.684487 18017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:07.676290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39209 (local address 127.0.98.62:43127)
0927 12:50:07.676418 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:07.676421 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:07.677513 (+ 1092us) server_negotiation.cc:407] Connection header received
0927 12:50:07.678996 (+ 1483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:07.679000 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:07.679052 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:07.679171 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:07.680231 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:07.680815 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:07.682197 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:07.682446 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:07.682537 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:07.683943 (+ 1406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:07.683946 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:07.683948 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:50:07.683963 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:07.683986 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:50:07.683995 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:07.684057 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:07.684264 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 12:50:07.684339 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 12:50:07.685441 18299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:07.676699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43127 (local address 127.0.98.2:39209)
0927 12:50:07.677364 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:07.677406 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 12:50:07.678690 (+ 1284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:50:07.679337 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:50:07.679345 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:50:07.679441 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:50:07.680051 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:07.680075 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:07.680975 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:07.680978 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:07.682008 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:07.682018 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:07.682609 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:07.682612 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:07.682854 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:07.683524 (+ 670us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 12:50:07.683526 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:50:07.683589 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 12:50:07.683685 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:50:07.684075 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:50:07.684084 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:50:07.684091 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 12:50:07.684233 (+ 142us) client_negotiation.cc:240] Negotiation successful
0927 12:50:07.684570 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":108,"threads_started":1}
I20260927 12:50:07.686224 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790513161160093-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790513161160093-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790513161160093-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43127
--builtin_ntp_servers=127.0.98.20:45005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 12:50:07.686635 18297 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43127
I20260927 12:50:07.686899 18297 heartbeater.cc:464] Registering TS with master...
I20260927 12:50:07.687522 18297 heartbeater.cc:511] Master 127.0.98.62:43127 requested a full tablet report, sending...
I20260927 12:50:07.688656 17978 ts_manager.cc:194] Registered new tserver with Master: 05b7b16ecb644f58882ca57898d09c43 (127.0.98.2:38015)
I20260927 12:50:07.689409 17978 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:39209
W20260927 12:50:07.801288 18300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:50:07.801568 18300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:50:07.801605 18300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:50:07.805660 18300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:50:07.805732 18300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:50:07.805795 18300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:50:07.810825 18300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790513161160093-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790513161160093-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18300
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790513161160093-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:50:07.811954 18300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:50:07.813333 18300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:50:07.822784 18308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:50:07.822788 18306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:50:07.822872 18305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:50:07.822938 18300 server_base.cc:1034] running on GCE node
I20260927 12:50:07.823431 18300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:50:07.824070 18300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:50:07.825245 18300 hybrid_clock.cc:648] HybridClock initialized: now 1790513407825215 us; error 39 us; skew 500 ppm
I20260927 12:50:07.827145 18300 webserver.cc:460] Webserver started at http://127.0.98.3:39061/ using document root <none> and password file <none>
I20260927 12:50:07.827724 18300 fs_manager.cc:359] Metadata directory not provided
I20260927 12:50:07.827786 18300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:50:07.827984 18300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:50:07.830088 18300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790513161160093-392-0/minicluster-data/ts-2/data/instance:
uuid: "51a59178f70c4d0480119313f6413e19"
format_stamp: "Formatted at 2026-09-27 12:50:07 on dist-test-slave-lkn4"
server_key: "aaeaaf80fe1d0599a690810e408e79d7"
server_key_iv: "aed3f81552f3bc0ba2f43e03d9e08f60"
server_key_version: "encryptionkey@0"
I20260927 12:50:07.830646 18300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790513161160093-392-0/minicluster-data/ts-2/wal/instance:
uuid: "51a59178f70c4d0480119313f6413e19"
format_stamp: "Formatted at 2026-09-27 12:50:07 on dist-test-slave-lkn4"
server_key: "aaeaaf80fe1d0599a690810e408e79d7"
server_key_iv: "aed3f81552f3bc0ba2f43e03d9e08f60"
server_key_version: "encryptionkey@0"
I20260927 12:50:07.834321 18300 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:50:07.836794 18314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:07.838114 18300 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:50:07.838217 18300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790513161160093-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790513161160093-392-0/minicluster-data/ts-2/wal
uuid: "51a59178f70c4d0480119313f6413e19"
format_stamp: "Formatted at 2026-09-27 12:50:07 on dist-test-slave-lkn4"
server_key: "aaeaaf80fe1d0599a690810e408e79d7"
server_key_iv: "aed3f81552f3bc0ba2f43e03d9e08f60"
server_key_version: "encryptionkey@0"
I20260927 12:50:07.838318 18300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790513161160093-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790513161160093-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790513161160093-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:50:07.885403 18300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:50:07.886190 18300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:50:07.886374 18300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:50:07.886997 18300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:50:07.887959 18300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:50:07.888002 18300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:07.888041 18300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:50:07.888057 18300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:07.896627 18300 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44849
I20260927 12:50:07.896661 18427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44849 every 8 connection(s)
I20260927 12:50:07.897548 18300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:50:07.903849 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 18300
I20260927 12:50:07.903956 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790513161160093-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:50:07.904223 392 external_mini_cluster.cc:1428] Setting key 80c085aad4372fb38cbaab246aa453fd
I20260927 12:50:07.907394 18017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:07.899470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35625 (local address 127.0.98.62:43127)
0927 12:50:07.899612 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:07.899615 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:07.900750 (+ 1135us) server_negotiation.cc:407] Connection header received
0927 12:50:07.902172 (+ 1422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:07.902176 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:07.902223 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:07.902296 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:07.903253 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:07.903812 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:07.905231 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:07.905521 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:07.905645 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:07.906976 (+ 1331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:07.906979 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:07.906981 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:50:07.906993 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:07.907018 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:50:07.907025 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:07.907087 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:07.907232 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 12:50:07.907284 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 12:50:07.908342 18430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:07.899929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43127 (local address 127.0.98.3:35625)
0927 12:50:07.900596 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:07.900637 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 12:50:07.901943 (+ 1306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:50:07.902447 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:50:07.902454 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:50:07.902549 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:50:07.903097 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:07.903109 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:07.903958 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:07.903962 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:07.905108 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:07.905120 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:07.905696 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:07.905700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:07.905898 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:07.906600 (+ 702us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 12:50:07.906601 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:50:07.906643 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 12:50:07.906709 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:50:07.907106 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:50:07.907114 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:50:07.907118 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 12:50:07.907212 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 12:50:07.907471 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":103,"threads_started":1}
I20260927 12:50:07.909579 18428 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43127
I20260927 12:50:07.909834 18428 heartbeater.cc:464] Registering TS with master...
I20260927 12:50:07.910336 18428 heartbeater.cc:511] Master 127.0.98.62:43127 requested a full tablet report, sending...
I20260927 12:50:07.911283 17978 ts_manager.cc:194] Registered new tserver with Master: 51a59178f70c4d0480119313f6413e19 (127.0.98.3:44849)
I20260927 12:50:07.911836 17978 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:35625
I20260927 12:50:07.918761 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 12:50:07.924710 18017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:07.920812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43788 (local address 127.0.98.62:43127)
0927 12:50:07.920930 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:07.920935 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:07.921076 (+ 141us) server_negotiation.cc:407] Connection header received
0927 12:50:07.921205 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:07.921208 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:07.921255 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:07.921349 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:07.921700 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:07.922390 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:07.923238 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:07.923453 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:07.923535 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:07.924303 (+ 768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:07.924306 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:07.924308 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:50:07.924321 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:07.924340 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:50:07.924358 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:07.924414 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:07.924542 (+ 128us) server_negotiation.cc:299] Negotiation successful
0927 12:50:07.924595 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 12:50:07.927670 17978 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43788:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:50:07.930819 17978 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:50:07.945628 18441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:07.939735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50974 (local address 127.0.98.1:43515)
0927 12:50:07.940188 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:07.940201 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:07.940583 (+ 382us) server_negotiation.cc:407] Connection header received
0927 12:50:07.940889 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:07.940896 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:07.941108 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:07.941263 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:50:07.941627 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:07.942884 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:07.944192 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:07.944752 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:07.944858 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:07.945013 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:07.945404 (+ 391us) server_negotiation.cc:299] Negotiation successful
0927 12:50:07.945491 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":131,"threads_started":1}
I20260927 12:50:07.945698 18442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:07.939564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43515 (local address 127.0.0.1:50974)
0927 12:50:07.940435 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:07.940484 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 12:50:07.940664 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:50:07.941354 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:50:07.941361 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:50:07.941383 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:50:07.941524 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:07.941531 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:07.942977 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:07.942981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:07.944107 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:07.944115 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:07.945065 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:07.945069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:07.945213 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:07.945265 (+ 52us) client_negotiation.cc:769] Sending connection context
0927 12:50:07.945374 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 12:50:07.945525 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700,"mutex_wait_us":31,"thread_start_us":91,"threads_started":1}
I20260927 12:50:07.946206 18440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:07.939930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44849 (local address 127.0.0.1:36214)
0927 12:50:07.940768 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:07.940779 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 12:50:07.940842 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:50:07.941593 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:50:07.941596 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:50:07.941611 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:50:07.941698 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:07.941703 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:07.943094 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:07.943097 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:07.944092 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:07.944100 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:07.945108 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:07.945111 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:07.945772 (+ 661us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:07.945920 (+ 148us) client_negotiation.cc:769] Sending connection context
0927 12:50:07.945988 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 12:50:07.946081 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":784,"mutex_wait_us":7,"thread_start_us":89,"threads_started":1}
I20260927 12:50:07.946422 18444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:07.940434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38278 (local address 127.0.98.2:38015)
0927 12:50:07.941020 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:07.941031 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:07.941052 (+ 21us) server_negotiation.cc:407] Connection header received
0927 12:50:07.941122 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:07.941127 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:07.941310 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:07.941452 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:50:07.941649 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:07.942512 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:07.944181 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:07.944995 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:07.945095 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:07.945273 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:07.945834 (+ 561us) server_negotiation.cc:299] Negotiation successful
0927 12:50:07.945983 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":100,"threads_started":1}
I20260927 12:50:07.946682 18443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:07.940363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36214 (local address 127.0.98.3:44849)
0927 12:50:07.941068 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:07.941078 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:07.941101 (+ 23us) server_negotiation.cc:407] Connection header received
0927 12:50:07.941169 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:07.941175 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:07.941343 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:07.941477 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:50:07.941870 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:07.942988 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:07.944215 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:07.944994 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:07.945101 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:07.945274 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:07.946337 (+ 1063us) server_negotiation.cc:299] Negotiation successful
0927 12:50:07.946495 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":82,"threads_started":1}
I20260927 12:50:07.945698 18439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:07.939732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38015 (local address 127.0.0.1:38278)
0927 12:50:07.940436 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:07.940486 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 12:50:07.940609 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:50:07.941432 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:50:07.941435 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:50:07.941451 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:50:07.941541 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:07.941547 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:07.942612 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:07.942615 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:07.944042 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:07.944053 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:07.945194 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:07.945197 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:07.945265 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:07.945283 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 12:50:07.945374 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 12:50:07.945525 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"mutex_wait_us":44,"thread_start_us":99,"threads_started":1}
I20260927 12:50:07.948762 18100 tablet_service.cc:1467] Processing CreateTablet for tablet d9e5cf96962a4755a2856feeba1d2e13 (DEFAULT_TABLE table=test-table [id=7cd60c8202b6406fb47877bf6a8439d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:50:07.949039 18362 tablet_service.cc:1467] Processing CreateTablet for tablet d9e5cf96962a4755a2856feeba1d2e13 (DEFAULT_TABLE table=test-table [id=7cd60c8202b6406fb47877bf6a8439d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:50:07.949684 18100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9e5cf96962a4755a2856feeba1d2e13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:50:07.949803 18362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9e5cf96962a4755a2856feeba1d2e13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:50:07.952991 18231 tablet_service.cc:1467] Processing CreateTablet for tablet d9e5cf96962a4755a2856feeba1d2e13 (DEFAULT_TABLE table=test-table [id=7cd60c8202b6406fb47877bf6a8439d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:50:07.954178 18231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9e5cf96962a4755a2856feeba1d2e13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:50:07.956321 18445 tablet_bootstrap.cc:492] T d9e5cf96962a4755a2856feeba1d2e13 P 53bebd9622f64e88bbc3d59a308b630b: Bootstrap starting.
I20260927 12:50:07.957764 18446 tablet_bootstrap.cc:492] T d9e5cf96962a4755a2856feeba1d2e13 P 51a59178f70c4d0480119313f6413e19: Bootstrap starting.
I20260927 12:50:07.958775 18445 tablet_bootstrap.cc:654] T d9e5cf96962a4755a2856feeba1d2e13 P 53bebd9622f64e88bbc3d59a308b630b: Neither blocks nor log segments found. Creating new log.
I20260927 12:50:07.959815 18445 log.cc:824] T d9e5cf96962a4755a2856feeba1d2e13 P 53bebd9622f64e88bbc3d59a308b630b: Log is configured to *not* fsync() on all Append() calls
I20260927 12:50:07.959921 18446 tablet_bootstrap.cc:654] T d9e5cf96962a4755a2856feeba1d2e13 P 51a59178f70c4d0480119313f6413e19: Neither blocks nor log segments found. Creating new log.
I20260927 12:50:07.960659 18446 log.cc:824] T d9e5cf96962a4755a2856feeba1d2e13 P 51a59178f70c4d0480119313f6413e19: Log is configured to *not* fsync() on all Append() calls
I20260927 12:50:07.961083 18448 tablet_bootstrap.cc:492] T d9e5cf96962a4755a2856feeba1d2e13 P 05b7b16ecb644f58882ca57898d09c43: Bootstrap starting.
I20260927 12:50:07.962001 18445 tablet_bootstrap.cc:492] T d9e5cf96962a4755a2856feeba1d2e13 P 53bebd9622f64e88bbc3d59a308b630b: No bootstrap required, opened a new log
I20260927 12:50:07.962234 18445 ts_tablet_manager.cc:1397] T d9e5cf96962a4755a2856feeba1d2e13 P 53bebd9622f64e88bbc3d59a308b630b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.002s
I20260927 12:50:07.963101 18446 tablet_bootstrap.cc:492] T d9e5cf96962a4755a2856feeba1d2e13 P 51a59178f70c4d0480119313f6413e19: No bootstrap required, opened a new log
I20260927 12:50:07.963310 18446 ts_tablet_manager.cc:1397] T d9e5cf96962a4755a2856feeba1d2e13 P 51a59178f70c4d0480119313f6413e19: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 12:50:07.963400 18448 tablet_bootstrap.cc:654] T d9e5cf96962a4755a2856feeba1d2e13 P 05b7b16ecb644f58882ca57898d09c43: Neither blocks nor log segments found. Creating new log.
I20260927 12:50:07.964273 18448 log.cc:824] T d9e5cf96962a4755a2856feeba1d2e13 P 05b7b16ecb644f58882ca57898d09c43: Log is configured to *not* fsync() on all Append() calls
I20260927 12:50:07.965906 18445 raft_consensus.cc:348] T d9e5cf96962a4755a2856feeba1d2e13 P 53bebd9622f64e88bbc3d59a308b630b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bebd9622f64e88bbc3d59a308b630b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43515 } } peers { permanent_uuid: "05b7b16ecb644f58882ca57898d09c43" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38015 } } peers { permanent_uuid: "51a59178f70c4d0480119313f6413e19" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44849 } }
I20260927 12:50:07.966195 18445 raft_consensus.cc:374] T d9e5cf96962a4755a2856feeba1d2e13 P 53bebd9622f64e88bbc3d59a308b630b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:50:07.966253 18445 raft_consensus.cc:729] T d9e5cf96962a4755a2856feeba1d2e13 P 53bebd9622f64e88bbc3d59a308b630b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53bebd9622f64e88bbc3d59a308b630b, State: Initialized, Role: FOLLOWER
I20260927 12:50:07.966683 18445 consensus_queue.cc:260] T d9e5cf96962a4755a2856feeba1d2e13 P 53bebd9622f64e88bbc3d59a308b630b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bebd9622f64e88bbc3d59a308b630b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43515 } } peers { permanent_uuid: "05b7b16ecb644f58882ca57898d09c43" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38015 } } peers { permanent_uuid: "51a59178f70c4d0480119313f6413e19" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44849 } }
I20260927 12:50:07.967381 18448 tablet_bootstrap.cc:492] T d9e5cf96962a4755a2856feeba1d2e13 P 05b7b16ecb644f58882ca57898d09c43: No bootstrap required, opened a new log
I20260927 12:50:07.967517 18166 heartbeater.cc:503] Master 127.0.98.62:43127 was elected leader, sending a full tablet report...
I20260927 12:50:07.967600 18448 ts_tablet_manager.cc:1397] T d9e5cf96962a4755a2856feeba1d2e13 P 05b7b16ecb644f58882ca57898d09c43: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260927 12:50:07.967613 18446 raft_consensus.cc:348] T d9e5cf96962a4755a2856feeba1d2e13 P 51a59178f70c4d0480119313f6413e19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bebd9622f64e88bbc3d59a308b630b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43515 } } peers { permanent_uuid: "05b7b16ecb644f58882ca57898d09c43" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38015 } } peers { permanent_uuid: "51a59178f70c4d0480119313f6413e19" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44849 } }
I20260927 12:50:07.967922 18446 raft_consensus.cc:374] T d9e5cf96962a4755a2856feeba1d2e13 P 51a59178f70c4d0480119313f6413e19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:50:07.967980 18446 raft_consensus.cc:729] T d9e5cf96962a4755a2856feeba1d2e13 P 51a59178f70c4d0480119313f6413e19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51a59178f70c4d0480119313f6413e19, State: Initialized, Role: FOLLOWER
I20260927 12:50:07.968464 18446 consensus_queue.cc:260] T d9e5cf96962a4755a2856feeba1d2e13 P 51a59178f70c4d0480119313f6413e19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bebd9622f64e88bbc3d59a308b630b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43515 } } peers { permanent_uuid: "05b7b16ecb644f58882ca57898d09c43" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38015 } } peers { permanent_uuid: "51a59178f70c4d0480119313f6413e19" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44849 } }
I20260927 12:50:07.969086 18445 ts_tablet_manager.cc:1428] T d9e5cf96962a4755a2856feeba1d2e13 P 53bebd9622f64e88bbc3d59a308b630b: Time spent starting tablet: real 0.007s user 0.003s sys 0.002s
I20260927 12:50:07.969524 18446 ts_tablet_manager.cc:1428] T d9e5cf96962a4755a2856feeba1d2e13 P 51a59178f70c4d0480119313f6413e19: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 12:50:07.969692 18428 heartbeater.cc:503] Master 127.0.98.62:43127 was elected leader, sending a full tablet report...
I20260927 12:50:07.972128 18448 raft_consensus.cc:348] T d9e5cf96962a4755a2856feeba1d2e13 P 05b7b16ecb644f58882ca57898d09c43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bebd9622f64e88bbc3d59a308b630b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43515 } } peers { permanent_uuid: "05b7b16ecb644f58882ca57898d09c43" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38015 } } peers { permanent_uuid: "51a59178f70c4d0480119313f6413e19" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44849 } }
I20260927 12:50:07.972460 18448 raft_consensus.cc:374] T d9e5cf96962a4755a2856feeba1d2e13 P 05b7b16ecb644f58882ca57898d09c43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:50:07.972527 18448 raft_consensus.cc:729] T d9e5cf96962a4755a2856feeba1d2e13 P 05b7b16ecb644f58882ca57898d09c43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05b7b16ecb644f58882ca57898d09c43, State: Initialized, Role: FOLLOWER
I20260927 12:50:07.973070 18448 consensus_queue.cc:260] T d9e5cf96962a4755a2856feeba1d2e13 P 05b7b16ecb644f58882ca57898d09c43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bebd9622f64e88bbc3d59a308b630b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43515 } } peers { permanent_uuid: "05b7b16ecb644f58882ca57898d09c43" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38015 } } peers { permanent_uuid: "51a59178f70c4d0480119313f6413e19" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44849 } }
I20260927 12:50:07.973824 18297 heartbeater.cc:503] Master 127.0.98.62:43127 was elected leader, sending a full tablet report...
I20260927 12:50:07.974071 18448 ts_tablet_manager.cc:1428] T d9e5cf96962a4755a2856feeba1d2e13 P 05b7b16ecb644f58882ca57898d09c43: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
W20260927 12:50:07.974670 18167 tablet.cc:2369] T d9e5cf96962a4755a2856feeba1d2e13 P 53bebd9622f64e88bbc3d59a308b630b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:50:08.028527 18452 raft_consensus.cc:482] T d9e5cf96962a4755a2856feeba1d2e13 P 51a59178f70c4d0480119313f6413e19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:50:08.028738 18452 raft_consensus.cc:504] T d9e5cf96962a4755a2856feeba1d2e13 P 51a59178f70c4d0480119313f6413e19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bebd9622f64e88bbc3d59a308b630b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43515 } } peers { permanent_uuid: "05b7b16ecb644f58882ca57898d09c43" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38015 } } peers { permanent_uuid: "51a59178f70c4d0480119313f6413e19" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44849 } }
I20260927 12:50:08.029996 18452 leader_election.cc:288] T d9e5cf96962a4755a2856feeba1d2e13 P 51a59178f70c4d0480119313f6413e19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53bebd9622f64e88bbc3d59a308b630b (127.0.98.1:43515), 05b7b16ecb644f58882ca57898d09c43 (127.0.98.2:38015)
I20260927 12:50:08.033736 18444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:08.030387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46935 (local address 127.0.98.2:38015)
0927 12:50:08.030505 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:08.030510 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:08.030527 (+ 17us) server_negotiation.cc:407] Connection header received
0927 12:50:08.030705 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:08.030709 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:08.030765 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:08.030834 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:50:08.031129 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:08.031648 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:08.032795 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:08.033195 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:08.033300 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:08.033364 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:08.033527 (+ 163us) server_negotiation.cc:299] Negotiation successful
0927 12:50:08.033606 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 12:50:08.033758 18430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:08.030322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38015 (local address 127.0.98.3:46935)
0927 12:50:08.030449 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:08.030464 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:50:08.030566 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:50:08.030857 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:50:08.030860 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:50:08.030907 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:50:08.031017 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:08.031023 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:08.031836 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:08.031839 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:08.032656 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:08.032665 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:08.033310 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:08.033313 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:08.033411 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:08.033434 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 12:50:08.033497 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 12:50:08.033571 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"spinlock_wait_cycles":7552}
I20260927 12:50:08.033758 18441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:08.030434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48223 (local address 127.0.98.1:43515)
0927 12:50:08.030545 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:08.030549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:08.030867 (+ 318us) server_negotiation.cc:407] Connection header received
0927 12:50:08.031007 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:08.031011 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:08.031052 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:08.031120 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:50:08.031363 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:08.031849 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:08.032859 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:08.033256 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:08.033347 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:08.033405 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:08.033557 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 12:50:08.033627 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 12:50:08.033776 18454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:08.030322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43515 (local address 127.0.98.3:48223)
0927 12:50:08.030776 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:08.030787 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 12:50:08.030910 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:50:08.031128 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:50:08.031131 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:50:08.031145 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:50:08.031267 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:08.031271 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:08.031944 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:08.031947 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:08.032746 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:08.032754 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:08.033362 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:08.033366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:08.033456 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:08.033474 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 12:50:08.033535 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 12:50:08.033600 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"mutex_wait_us":114,"spinlock_wait_cycles":26880,"thread_start_us":134,"threads_started":1}
I20260927 12:50:08.034513 18251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9e5cf96962a4755a2856feeba1d2e13" candidate_uuid: "51a59178f70c4d0480119313f6413e19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b7b16ecb644f58882ca57898d09c43" is_pre_election: true
I20260927 12:50:08.034509 18120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9e5cf96962a4755a2856feeba1d2e13" candidate_uuid: "51a59178f70c4d0480119313f6413e19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53bebd9622f64e88bbc3d59a308b630b" is_pre_election: true
I20260927 12:50:08.034788 18120 raft_consensus.cc:2454] T d9e5cf96962a4755a2856feeba1d2e13 P 53bebd9622f64e88bbc3d59a308b630b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51a59178f70c4d0480119313f6413e19 in term 0.
I20260927 12:50:08.034786 18251 raft_consensus.cc:2454] T d9e5cf96962a4755a2856feeba1d2e13 P 05b7b16ecb644f58882ca57898d09c43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51a59178f70c4d0480119313f6413e19 in term 0.
I20260927 12:50:08.035254 18315 leader_election.cc:302] T d9e5cf96962a4755a2856feeba1d2e13 P 51a59178f70c4d0480119313f6413e19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51a59178f70c4d0480119313f6413e19, 53bebd9622f64e88bbc3d59a308b630b; no voters:
I20260927 12:50:08.035512 18452 raft_consensus.cc:2781] T d9e5cf96962a4755a2856feeba1d2e13 P 51a59178f70c4d0480119313f6413e19 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:50:08.035591 18452 raft_consensus.cc:482] T d9e5cf96962a4755a2856feeba1d2e13 P 51a59178f70c4d0480119313f6413e19 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:50:08.035622 18452 raft_consensus.cc:3037] T d9e5cf96962a4755a2856feeba1d2e13 P 51a59178f70c4d0480119313f6413e19 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:50:08.036659 18452 raft_consensus.cc:504] T d9e5cf96962a4755a2856feeba1d2e13 P 51a59178f70c4d0480119313f6413e19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bebd9622f64e88bbc3d59a308b630b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43515 } } peers { permanent_uuid: "05b7b16ecb644f58882ca57898d09c43" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38015 } } peers { permanent_uuid: "51a59178f70c4d0480119313f6413e19" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44849 } }
I20260927 12:50:08.037004 18452 leader_election.cc:288] T d9e5cf96962a4755a2856feeba1d2e13 P 51a59178f70c4d0480119313f6413e19 [CANDIDATE]: Term 1 election: Requested vote from peers 53bebd9622f64e88bbc3d59a308b630b (127.0.98.1:43515), 05b7b16ecb644f58882ca57898d09c43 (127.0.98.2:38015)
I20260927 12:50:08.037351 18251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9e5cf96962a4755a2856feeba1d2e13" candidate_uuid: "51a59178f70c4d0480119313f6413e19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b7b16ecb644f58882ca57898d09c43"
I20260927 12:50:08.037393 18120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9e5cf96962a4755a2856feeba1d2e13" candidate_uuid: "51a59178f70c4d0480119313f6413e19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53bebd9622f64e88bbc3d59a308b630b"
I20260927 12:50:08.037485 18251 raft_consensus.cc:3037] T d9e5cf96962a4755a2856feeba1d2e13 P 05b7b16ecb644f58882ca57898d09c43 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:50:08.037523 18120 raft_consensus.cc:3037] T d9e5cf96962a4755a2856feeba1d2e13 P 53bebd9622f64e88bbc3d59a308b630b [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:50:08.038787 18251 raft_consensus.cc:2454] T d9e5cf96962a4755a2856feeba1d2e13 P 05b7b16ecb644f58882ca57898d09c43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51a59178f70c4d0480119313f6413e19 in term 1.
I20260927 12:50:08.038795 18120 raft_consensus.cc:2454] T d9e5cf96962a4755a2856feeba1d2e13 P 53bebd9622f64e88bbc3d59a308b630b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51a59178f70c4d0480119313f6413e19 in term 1.
I20260927 12:50:08.039119 18317 leader_election.cc:302] T d9e5cf96962a4755a2856feeba1d2e13 P 51a59178f70c4d0480119313f6413e19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b7b16ecb644f58882ca57898d09c43, 51a59178f70c4d0480119313f6413e19; no voters:
I20260927 12:50:08.039351 18452 raft_consensus.cc:2781] T d9e5cf96962a4755a2856feeba1d2e13 P 51a59178f70c4d0480119313f6413e19 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:50:08.039556 18452 raft_consensus.cc:686] T d9e5cf96962a4755a2856feeba1d2e13 P 51a59178f70c4d0480119313f6413e19 [term 1 LEADER]: Becoming Leader. State: Replica: 51a59178f70c4d0480119313f6413e19, State: Running, Role: LEADER
I20260927 12:50:08.039866 18452 consensus_queue.cc:237] T d9e5cf96962a4755a2856feeba1d2e13 P 51a59178f70c4d0480119313f6413e19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bebd9622f64e88bbc3d59a308b630b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43515 } } peers { permanent_uuid: "05b7b16ecb644f58882ca57898d09c43" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38015 } } peers { permanent_uuid: "51a59178f70c4d0480119313f6413e19" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44849 } }
I20260927 12:50:08.043062 17977 catalog_manager.cc:5495] T d9e5cf96962a4755a2856feeba1d2e13 P 51a59178f70c4d0480119313f6413e19 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51a59178f70c4d0480119313f6413e19 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "51a59178f70c4d0480119313f6413e19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53bebd9622f64e88bbc3d59a308b630b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05b7b16ecb644f58882ca57898d09c43" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51a59178f70c4d0480119313f6413e19" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44849 } health_report { overall_health: HEALTHY } } }
I20260927 12:50:08.073431 18443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:08.069957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36220 (local address 127.0.98.3:44849)
0927 12:50:08.070120 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:08.070124 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:08.070137 (+ 13us) server_negotiation.cc:407] Connection header received
0927 12:50:08.070237 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:08.070240 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:08.070297 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:08.070381 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:50:08.070571 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:08.071127 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:08.071981 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:08.072192 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:08.072275 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:08.072537 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:50:08.072972 (+ 435us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:50:08.073062 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:08.073223 (+ 161us) server_negotiation.cc:299] Negotiation successful
0927 12:50:08.073278 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 12:50:08.082448 18251 raft_consensus.cc:1261] T d9e5cf96962a4755a2856feeba1d2e13 P 05b7b16ecb644f58882ca57898d09c43 [term 1 FOLLOWER]: Refusing update from remote peer 51a59178f70c4d0480119313f6413e19: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:50:08.084030 18120 raft_consensus.cc:1261] T d9e5cf96962a4755a2856feeba1d2e13 P 53bebd9622f64e88bbc3d59a308b630b [term 1 FOLLOWER]: Refusing update from remote peer 51a59178f70c4d0480119313f6413e19: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:50:08.084193 18455 consensus_queue.cc:1035] T d9e5cf96962a4755a2856feeba1d2e13 P 51a59178f70c4d0480119313f6413e19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05b7b16ecb644f58882ca57898d09c43" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:50:08.084762 18452 consensus_queue.cc:1035] T d9e5cf96962a4755a2856feeba1d2e13 P 51a59178f70c4d0480119313f6413e19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53bebd9622f64e88bbc3d59a308b630b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:50:08.092630 18458 mvcc.cc:205] Tried to move back new op lower bound from 7333942919486935040 to 7333942919334170624. Current Snapshot: MvccSnapshot[applied={T|T < 7333942919486935040}]
I20260927 12:50:08.094537 18460 mvcc.cc:205] Tried to move back new op lower bound from 7333942919486935040 to 7333942919334170624. Current Snapshot: MvccSnapshot[applied={T|T < 7333942919486935040}]
I20260927 12:50:08.097777 18459 mvcc.cc:205] Tried to move back new op lower bound from 7333942919486935040 to 7333942919334170624. Current Snapshot: MvccSnapshot[applied={T|T < 7333942919486935040}]
I20260927 12:50:08.104112 17977 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:50:08.104308 17977 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:50:08.107263 17977 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0e42f9e22631430b8cbf3ed3a72ec64b: Sending DeleteTablet for 3 replicas of tablet d9e5cf96962a4755a2856feeba1d2e13
I20260927 12:50:08.108206 18100 tablet_service.cc:1514] Processing DeleteTablet for tablet d9e5cf96962a4755a2856feeba1d2e13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:50:08 UTC) from {username='slave'} at 127.0.0.1:50974
I20260927 12:50:08.108420 18362 tablet_service.cc:1514] Processing DeleteTablet for tablet d9e5cf96962a4755a2856feeba1d2e13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:50:08 UTC) from {username='slave'} at 127.0.0.1:36214
I20260927 12:50:08.109241 18231 tablet_service.cc:1514] Processing DeleteTablet for tablet d9e5cf96962a4755a2856feeba1d2e13 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:50:08 UTC) from {username='slave'} at 127.0.0.1:38278
I20260927 12:50:08.109757 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 18038
I20260927 12:50:08.109853 18469 tablet_replica.cc:330] T d9e5cf96962a4755a2856feeba1d2e13 P 05b7b16ecb644f58882ca57898d09c43: stopping tablet replica
I20260927 12:50:08.110342 18469 raft_consensus.cc:2229] T d9e5cf96962a4755a2856feeba1d2e13 P 05b7b16ecb644f58882ca57898d09c43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:50:08.110773 18469 raft_consensus.cc:2258] T d9e5cf96962a4755a2856feeba1d2e13 P 05b7b16ecb644f58882ca57898d09c43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:50:08.112542 18468 tablet_replica.cc:330] T d9e5cf96962a4755a2856feeba1d2e13 P 51a59178f70c4d0480119313f6413e19: stopping tablet replica
I20260927 12:50:08.112572 18469 ts_tablet_manager.cc:1905] T d9e5cf96962a4755a2856feeba1d2e13 P 05b7b16ecb644f58882ca57898d09c43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:50:08.112910 18468 raft_consensus.cc:2229] T d9e5cf96962a4755a2856feeba1d2e13 P 51a59178f70c4d0480119313f6413e19 [term 1 LEADER]: Raft consensus shutting down.
I20260927 12:50:08.113328 18468 raft_consensus.cc:2258] T d9e5cf96962a4755a2856feeba1d2e13 P 51a59178f70c4d0480119313f6413e19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:50:08.114944 18468 ts_tablet_manager.cc:1905] T d9e5cf96962a4755a2856feeba1d2e13 P 51a59178f70c4d0480119313f6413e19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:50:08.115737 18469 ts_tablet_manager.cc:1918] T d9e5cf96962a4755a2856feeba1d2e13 P 05b7b16ecb644f58882ca57898d09c43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:50:08.115901 18469 log.cc:1196] T d9e5cf96962a4755a2856feeba1d2e13 P 05b7b16ecb644f58882ca57898d09c43: Deleting WAL directory at /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790513161160093-392-0/minicluster-data/ts-1/wal/wals/d9e5cf96962a4755a2856feeba1d2e13
I20260927 12:50:08.116284 18469 ts_tablet_manager.cc:1939] T d9e5cf96962a4755a2856feeba1d2e13 P 05b7b16ecb644f58882ca57898d09c43: Deleting consensus metadata
I20260927 12:50:08.117518 18468 ts_tablet_manager.cc:1918] T d9e5cf96962a4755a2856feeba1d2e13 P 51a59178f70c4d0480119313f6413e19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:50:08.117693 17964 catalog_manager.cc:4841] TS 05b7b16ecb644f58882ca57898d09c43 (127.0.98.2:38015): tablet d9e5cf96962a4755a2856feeba1d2e13 (table test-table [id=7cd60c8202b6406fb47877bf6a8439d3]) successfully deleted
I20260927 12:50:08.117944 18468 log.cc:1196] T d9e5cf96962a4755a2856feeba1d2e13 P 51a59178f70c4d0480119313f6413e19: Deleting WAL directory at /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790513161160093-392-0/minicluster-data/ts-2/wal/wals/d9e5cf96962a4755a2856feeba1d2e13
W20260927 12:50:08.118005 17962 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43515 (error 108)
I20260927 12:50:08.118259 18468 ts_tablet_manager.cc:1939] T d9e5cf96962a4755a2856feeba1d2e13 P 51a59178f70c4d0480119313f6413e19: Deleting consensus metadata
I20260927 12:50:08.119131 18439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:08.118814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43515 (local address 127.0.0.1:50980)
0927 12:50:08.118966 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:08.119041 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 12:50:08.119131 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 18169
I20260927 12:50:08.120487 17963 catalog_manager.cc:4841] TS 51a59178f70c4d0480119313f6413e19 (127.0.98.3:44849): tablet d9e5cf96962a4755a2856feeba1d2e13 (table test-table [id=7cd60c8202b6406fb47877bf6a8439d3]) successfully deleted
W20260927 12:50:08.120528 17962 catalog_manager.cc:4584] TS 53bebd9622f64e88bbc3d59a308b630b (127.0.98.1:43515): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d9e5cf96962a4755a2856feeba1d2e13: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43515: connect: Connection refused (error 111)
I20260927 12:50:08.124715 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 18300
I20260927 12:50:08.129997 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 17947
2026-09-27T12:50:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1109 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:50:08 dist-test-slave-lkn4 krb5kdc[18473](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:50:08 dist-test-slave-lkn4 krb5kdc[18473](info): set up 2 sockets
Sep 27 12:50:08 dist-test-slave-lkn4 krb5kdc[18473](info): commencing operation
krb5kdc: starting...
W20260927 12:50:09.085800 392 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/krb5kdc/test-user.keytab.
Sep 27 12:50:09 dist-test-slave-lkn4 krb5kdc[18473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:50:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:50:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:50:09.209539 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34023
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34023
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:50:09.320928 18486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:50:09.321241 18486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:50:09.321283 18486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:50:09.325493 18486 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:50:09.325570 18486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:50:09.325589 18486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:50:09.325605 18486 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:50:09.325618 18486 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:50:09.331228 18486 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18486
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:50:09.332391 18486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:50:09.333441 18486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:50:09.342242 18491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:50:09.342252 18492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:50:09.342314 18486 server_base.cc:1034] running on GCE node
W20260927 12:50:09.342218 18494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:50:09.343017 18486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:50:09.344125 18486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:50:09.345300 18486 hybrid_clock.cc:648] HybridClock initialized: now 1790513409345276 us; error 45 us; skew 500 ppm
Sep 27 12:50:09 dist-test-slave-lkn4 krb5kdc[18473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:50:09.348063 18486 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:50:09.349032 18486 webserver.cc:460] Webserver started at http://127.0.98.62:38473/ using document root <none> and password file <none>
I20260927 12:50:09.349584 18486 fs_manager.cc:359] Metadata directory not provided
I20260927 12:50:09.349649 18486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:50:09.349848 18486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:50:09.351974 18486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/minicluster-data/master-0/data/instance:
uuid: "e0fc18da404e4742901121732eeb27a0"
format_stamp: "Formatted at 2026-09-27 12:50:09 on dist-test-slave-lkn4"
server_key: "55c639a1eb3d5d8c4030ec2879c4ee31"
server_key_iv: "45e84e03714eeaea1bb6a235eecc89bb"
server_key_version: "encryptionkey@0"
I20260927 12:50:09.352537 18486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/minicluster-data/master-0/wal/instance:
uuid: "e0fc18da404e4742901121732eeb27a0"
format_stamp: "Formatted at 2026-09-27 12:50:09 on dist-test-slave-lkn4"
server_key: "55c639a1eb3d5d8c4030ec2879c4ee31"
server_key_iv: "45e84e03714eeaea1bb6a235eecc89bb"
server_key_version: "encryptionkey@0"
I20260927 12:50:09.356482 18486 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 12:50:09.359067 18501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:09.360445 18486 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:50:09.360570 18486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/minicluster-data/master-0/wal
uuid: "e0fc18da404e4742901121732eeb27a0"
format_stamp: "Formatted at 2026-09-27 12:50:09 on dist-test-slave-lkn4"
server_key: "55c639a1eb3d5d8c4030ec2879c4ee31"
server_key_iv: "45e84e03714eeaea1bb6a235eecc89bb"
server_key_version: "encryptionkey@0"
I20260927 12:50:09.360677 18486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:50:09.380689 18486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:50:09.384291 18486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:50:09.384486 18486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:50:09.392068 18486 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34023
I20260927 12:50:09.392081 18553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34023 every 8 connection(s)
I20260927 12:50:09.393090 18486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:50:09.396042 18554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:50:09.396118 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 18486
I20260927 12:50:09.396245 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/minicluster-data/master-0/wal/instance
I20260927 12:50:09.396502 392 external_mini_cluster.cc:1428] Setting key 7fec138bc11777a66a1ac60253eec41b
I20260927 12:50:09.403501 18554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0fc18da404e4742901121732eeb27a0: Bootstrap starting.
I20260927 12:50:09.406217 18554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0fc18da404e4742901121732eeb27a0: Neither blocks nor log segments found. Creating new log.
Sep 27 12:50:09 dist-test-slave-lkn4 krb5kdc[18473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:50:09.407069 18554 log.cc:824] T 00000000000000000000000000000000 P e0fc18da404e4742901121732eeb27a0: Log is configured to *not* fsync() on all Append() calls
I20260927 12:50:09.409480 18554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0fc18da404e4742901121732eeb27a0: No bootstrap required, opened a new log
I20260927 12:50:09.412066 18557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:09.397934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46500 (local address 127.0.98.62:34023)
0927 12:50:09.398402 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:09.398421 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:09.398465 (+ 44us) server_negotiation.cc:407] Connection header received
0927 12:50:09.399433 (+ 968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:09.399462 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:09.399949 (+ 487us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:09.400359 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:09.401198 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:09.402762 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:09.403661 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:09.404348 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:09.404512 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:09.407207 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:09.407215 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:09.407233 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:50:09.407266 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:09.409222 (+ 1956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:09.409664 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:09.409670 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:09.409675 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:09.409794 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:09.410146 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:09.410149 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:09.410150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:09.410557 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:09.410727 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:09.411004 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 12:50:09.411323 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":139,"threads_started":1}
I20260927 12:50:09.413424 18554 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0fc18da404e4742901121732eeb27a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0fc18da404e4742901121732eeb27a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34023 } }
I20260927 12:50:09.413647 18554 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0fc18da404e4742901121732eeb27a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:50:09.413710 18554 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0fc18da404e4742901121732eeb27a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0fc18da404e4742901121732eeb27a0, State: Initialized, Role: FOLLOWER
I20260927 12:50:09.414162 18554 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0fc18da404e4742901121732eeb27a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0fc18da404e4742901121732eeb27a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34023 } }
I20260927 12:50:09.414301 18554 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0fc18da404e4742901121732eeb27a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:50:09.414390 18554 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0fc18da404e4742901121732eeb27a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:50:09.414481 18554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0fc18da404e4742901121732eeb27a0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:50:09.415397 18554 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0fc18da404e4742901121732eeb27a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0fc18da404e4742901121732eeb27a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34023 } }
I20260927 12:50:09.415725 18554 leader_election.cc:302] T 00000000000000000000000000000000 P e0fc18da404e4742901121732eeb27a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0fc18da404e4742901121732eeb27a0; no voters:
I20260927 12:50:09.416023 18554 leader_election.cc:288] T 00000000000000000000000000000000 P e0fc18da404e4742901121732eeb27a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:50:09.416193 18559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0fc18da404e4742901121732eeb27a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:50:09.416401 18559 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0fc18da404e4742901121732eeb27a0 [term 1 LEADER]: Becoming Leader. State: Replica: e0fc18da404e4742901121732eeb27a0, State: Running, Role: LEADER
I20260927 12:50:09.416747 18559 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0fc18da404e4742901121732eeb27a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0fc18da404e4742901121732eeb27a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34023 } }
I20260927 12:50:09.417238 18554 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0fc18da404e4742901121732eeb27a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:50:09.418499 18561 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0fc18da404e4742901121732eeb27a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0fc18da404e4742901121732eeb27a0. Latest consensus state: current_term: 1 leader_uuid: "e0fc18da404e4742901121732eeb27a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0fc18da404e4742901121732eeb27a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34023 } } }
I20260927 12:50:09.418622 18561 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0fc18da404e4742901121732eeb27a0 [sys.catalog]: This master's current role is: LEADER
I20260927 12:50:09.418605 18560 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0fc18da404e4742901121732eeb27a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0fc18da404e4742901121732eeb27a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0fc18da404e4742901121732eeb27a0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34023 } } }
I20260927 12:50:09.418731 18560 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0fc18da404e4742901121732eeb27a0 [sys.catalog]: This master's current role is: LEADER
I20260927 12:50:09.419044 18568 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:50:09.422399 18568 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:50:09.428642 18568 catalog_manager.cc:1347] Generated new cluster ID: 711cf1f4266d41c4bf8794d5d2325013
I20260927 12:50:09.428738 18568 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:50:09.435130 18568 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:50:09.436445 18568 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:50:09.442802 18568 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0fc18da404e4742901121732eeb27a0: Generated new TSK 0
I20260927 12:50:09.443768 18568 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 12:50:09.480964 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34023
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:50:09.595850 18580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:50:09.596154 18580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:50:09.596213 18580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:50:09.600657 18580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:50:09.600757 18580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:50:09.600848 18580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:50:09.606448 18580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:50:09.607627 18580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:50:09.608671 18580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:50:09.618587 18588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:50:09.618623 18585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:50:09.618770 18580 server_base.cc:1034] running on GCE node
W20260927 12:50:09.618963 18586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:50:09.619400 18580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:50:09.620137 18580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:50:09.621301 18580 hybrid_clock.cc:648] HybridClock initialized: now 1790513409621279 us; error 38 us; skew 500 ppm
Sep 27 12:50:09 dist-test-slave-lkn4 krb5kdc[18473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:50:09.624135 18580 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 12:50:09.625087 18580 webserver.cc:460] Webserver started at http://127.0.98.1:38823/ using document root <none> and password file <none>
I20260927 12:50:09.625648 18580 fs_manager.cc:359] Metadata directory not provided
I20260927 12:50:09.625708 18580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:50:09.625881 18580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:50:09.628060 18580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/minicluster-data/ts-0/data/instance:
uuid: "74c6cb864c124739b0dc7e6e8b2cf282"
format_stamp: "Formatted at 2026-09-27 12:50:09 on dist-test-slave-lkn4"
server_key: "aad47d4a75b1d5d6e90e74dda465b306"
server_key_iv: "ddd1f21cf6b82740fd0bdb58e3d1f230"
server_key_version: "encryptionkey@0"
I20260927 12:50:09.628584 18580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/minicluster-data/ts-0/wal/instance:
uuid: "74c6cb864c124739b0dc7e6e8b2cf282"
format_stamp: "Formatted at 2026-09-27 12:50:09 on dist-test-slave-lkn4"
server_key: "aad47d4a75b1d5d6e90e74dda465b306"
server_key_iv: "ddd1f21cf6b82740fd0bdb58e3d1f230"
server_key_version: "encryptionkey@0"
I20260927 12:50:09.632434 18580 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 12:50:09.635039 18595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:09.636825 18580 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:50:09.636961 18580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/minicluster-data/ts-0/wal
uuid: "74c6cb864c124739b0dc7e6e8b2cf282"
format_stamp: "Formatted at 2026-09-27 12:50:09 on dist-test-slave-lkn4"
server_key: "aad47d4a75b1d5d6e90e74dda465b306"
server_key_iv: "ddd1f21cf6b82740fd0bdb58e3d1f230"
server_key_version: "encryptionkey@0"
I20260927 12:50:09.637090 18580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:50:09.655249 18580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:50:09.658841 18580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:50:09.659042 18580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:50:09.659670 18580 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:50:09.660534 18580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:50:09.660584 18580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:09.660638 18580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:50:09.660668 18580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:09.669958 18580 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44439
I20260927 12:50:09.669983 18708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44439 every 8 connection(s)
I20260927 12:50:09.670920 18580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:50:09.678272 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 18580
I20260927 12:50:09.678406 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:50:09.678773 392 external_mini_cluster.cc:1428] Setting key 80fe57605f9bfffcc3245ef78e4f992c
Sep 27 12:50:09 dist-test-slave-lkn4 krb5kdc[18473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:50:09.685530 18557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:09.672750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42859 (local address 127.0.98.62:34023)
0927 12:50:09.672921 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:09.672926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:09.673798 (+ 872us) server_negotiation.cc:407] Connection header received
0927 12:50:09.674739 (+ 941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:09.674742 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:09.674795 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:09.674870 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:09.676043 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:09.676628 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:09.677787 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:09.678104 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:09.678226 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:09.680948 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:09.680952 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:09.680955 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:50:09.680984 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:09.682692 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:09.683356 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:09.683360 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:09.683362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:09.683462 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:09.683944 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:09.683948 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:09.683950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:09.684384 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:09.684553 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:09.685193 (+ 640us) server_negotiation.cc:299] Negotiation successful
0927 12:50:09.685341 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 12:50:09.686754 18711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:09.673166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34023 (local address 127.0.98.1:42859)
0927 12:50:09.673631 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:09.673676 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 12:50:09.674530 (+ 854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:50:09.675031 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:50:09.675038 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:50:09.675395 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:50:09.675889 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:09.675902 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:09.676737 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:09.676739 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:09.677671 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:09.677680 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:09.678294 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:09.678297 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:09.678488 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:09.679107 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:50:09.679124 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:50:09.680775 (+ 1651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:50:09.682821 (+ 2046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:50:09.682832 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:50:09.682853 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:50:09.683233 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:50:09.683578 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:50:09.683582 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:50:09.683584 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:50:09.683779 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:50:09.684628 (+ 849us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:50:09.684635 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:50:09.684923 (+ 288us) client_negotiation.cc:769] Sending connection context
0927 12:50:09.685135 (+ 212us) client_negotiation.cc:240] Negotiation successful
0927 12:50:09.685532 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":116,"threads_started":1}
I20260927 12:50:09.688169 18709 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34023
I20260927 12:50:09.688423 18709 heartbeater.cc:464] Registering TS with master...
I20260927 12:50:09.689011 18709 heartbeater.cc:511] Master 127.0.98.62:34023 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 12:50:09.690848 18518 ts_manager.cc:194] Registered new tserver with Master: 74c6cb864c124739b0dc7e6e8b2cf282 (127.0.98.1:44439)
I20260927 12:50:09.692209 18518 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42859
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 12:50:09.709883 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34023
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:50:09.824579 18714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:50:09.824903 18714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:50:09.824966 18714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:50:09.829221 18714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:50:09.829310 18714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:50:09.829398 18714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:50:09.834988 18714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:50:09.836114 18714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:50:09.837146 18714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 12:50:09.847380 18714 server_base.cc:1034] running on GCE node
W20260927 12:50:09.847323 18719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:50:09.847326 18720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:50:09.847362 18722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:50:09.848127 18714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:50:09.848773 18714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:50:09.849943 18714 hybrid_clock.cc:648] HybridClock initialized: now 1790513409849916 us; error 42 us; skew 500 ppm
Sep 27 12:50:09 dist-test-slave-lkn4 krb5kdc[18473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:50:09.856449 18714 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 12:50:09.857477 18714 webserver.cc:460] Webserver started at http://127.0.98.2:39637/ using document root <none> and password file <none>
I20260927 12:50:09.858172 18714 fs_manager.cc:359] Metadata directory not provided
I20260927 12:50:09.858245 18714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:50:09.858459 18714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:50:09.860633 18714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/minicluster-data/ts-1/data/instance:
uuid: "095bcd5daf9e4fe6ba7722f0f8853b99"
format_stamp: "Formatted at 2026-09-27 12:50:09 on dist-test-slave-lkn4"
server_key: "b13df179267ddd1ac093a899f52064e3"
server_key_iv: "08c73465a4e89d427912ccadc59a2114"
server_key_version: "encryptionkey@0"
I20260927 12:50:09.861171 18714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/minicluster-data/ts-1/wal/instance:
uuid: "095bcd5daf9e4fe6ba7722f0f8853b99"
format_stamp: "Formatted at 2026-09-27 12:50:09 on dist-test-slave-lkn4"
server_key: "b13df179267ddd1ac093a899f52064e3"
server_key_iv: "08c73465a4e89d427912ccadc59a2114"
server_key_version: "encryptionkey@0"
I20260927 12:50:09.865211 18714 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 12:50:09.867817 18729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:09.869140 18714 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:50:09.869268 18714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/minicluster-data/ts-1/wal
uuid: "095bcd5daf9e4fe6ba7722f0f8853b99"
format_stamp: "Formatted at 2026-09-27 12:50:09 on dist-test-slave-lkn4"
server_key: "b13df179267ddd1ac093a899f52064e3"
server_key_iv: "08c73465a4e89d427912ccadc59a2114"
server_key_version: "encryptionkey@0"
I20260927 12:50:09.869370 18714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:50:09.882711 18714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:50:09.886269 18714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:50:09.886456 18714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:50:09.887089 18714 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:50:09.887915 18714 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:50:09.887964 18714 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:09.888016 18714 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:50:09.888046 18714 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:09.896600 18714 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44675
I20260927 12:50:09.896672 18842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44675 every 8 connection(s)
I20260927 12:50:09.897518 18714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:50:09.906539 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 18714
I20260927 12:50:09.906675 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/minicluster-data/ts-1/wal/instance
Sep 27 12:50:09 dist-test-slave-lkn4 krb5kdc[18473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:50:09.907050 392 external_mini_cluster.cc:1428] Setting key 9b17db530c57f730eab982b3df0a4ec9
I20260927 12:50:09.911088 18557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:09.899422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35169 (local address 127.0.98.62:34023)
0927 12:50:09.899589 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:09.899593 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:09.900506 (+ 913us) server_negotiation.cc:407] Connection header received
0927 12:50:09.901479 (+ 973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:09.901482 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:09.901536 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:09.901611 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:09.902890 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:09.903414 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:09.904552 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:09.904775 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:09.904863 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:09.907550 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:09.907555 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:09.907557 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:50:09.907590 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:09.909011 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:09.909582 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:09.909585 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:09.909586 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:09.909655 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:09.909980 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:09.909982 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:09.909983 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:09.910207 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:09.910314 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:09.910797 (+ 483us) server_negotiation.cc:299] Negotiation successful
0927 12:50:09.910919 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 12:50:09.911916 18845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:09.899873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34023 (local address 127.0.98.2:35169)
0927 12:50:09.900347 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:09.900390 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 12:50:09.901267 (+ 877us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:50:09.901818 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:50:09.901826 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:50:09.902248 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:50:09.902741 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:09.902755 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:09.903531 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:09.903534 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:09.904428 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:09.904438 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:09.904915 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:09.904919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:09.905090 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:09.905662 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:50:09.905679 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:50:09.907347 (+ 1668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:50:09.909124 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:50:09.909131 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:50:09.909148 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:50:09.909499 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:50:09.909724 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:50:09.909727 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:50:09.909729 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:50:09.909906 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:50:09.910314 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:50:09.910320 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:50:09.910569 (+ 249us) client_negotiation.cc:769] Sending connection context
0927 12:50:09.910729 (+ 160us) client_negotiation.cc:240] Negotiation successful
0927 12:50:09.910999 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":113,"threads_started":1}
I20260927 12:50:09.913051 18843 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34023
I20260927 12:50:09.913290 18843 heartbeater.cc:464] Registering TS with master...
I20260927 12:50:09.913789 18843 heartbeater.cc:511] Master 127.0.98.62:34023 requested a full tablet report, sending...
I20260927 12:50:09.914944 18518 ts_manager.cc:194] Registered new tserver with Master: 095bcd5daf9e4fe6ba7722f0f8853b99 (127.0.98.2:44675)
I20260927 12:50:09.915517 18518 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35169
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 12:50:09.937144 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34023
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:50:10.049512 18848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:50:10.049798 18848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:50:10.049842 18848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:50:10.053956 18848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:50:10.054033 18848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:50:10.054147 18848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:50:10.059916 18848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:50:10.060966 18848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:50:10.061957 18848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:50:10.072322 18854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:50:10.072335 18853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:50:10.072403 18848 server_base.cc:1034] running on GCE node
W20260927 12:50:10.072321 18856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:50:10.073065 18848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:50:10.073755 18848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:50:10.074924 18848 hybrid_clock.cc:648] HybridClock initialized: now 1790513410074893 us; error 43 us; skew 500 ppm
Sep 27 12:50:10 dist-test-slave-lkn4 krb5kdc[18473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513410, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:50:10.077776 18848 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 12:50:10.078748 18848 webserver.cc:460] Webserver started at http://127.0.98.3:33971/ using document root <none> and password file <none>
I20260927 12:50:10.079308 18848 fs_manager.cc:359] Metadata directory not provided
I20260927 12:50:10.079375 18848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:50:10.079571 18848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:50:10.081648 18848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/minicluster-data/ts-2/data/instance:
uuid: "51fa4c129b74457abb540ff8c84d3b65"
format_stamp: "Formatted at 2026-09-27 12:50:10 on dist-test-slave-lkn4"
server_key: "e513f8a03fed4c7bc8fe8f099fcaa6af"
server_key_iv: "c70960d2b1ec407a2815dbcdcfe82f56"
server_key_version: "encryptionkey@0"
I20260927 12:50:10.082263 18848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/minicluster-data/ts-2/wal/instance:
uuid: "51fa4c129b74457abb540ff8c84d3b65"
format_stamp: "Formatted at 2026-09-27 12:50:10 on dist-test-slave-lkn4"
server_key: "e513f8a03fed4c7bc8fe8f099fcaa6af"
server_key_iv: "c70960d2b1ec407a2815dbcdcfe82f56"
server_key_version: "encryptionkey@0"
I20260927 12:50:10.086064 18848 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 12:50:10.088513 18863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:10.089830 18848 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:50:10.089941 18848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/minicluster-data/ts-2/wal
uuid: "51fa4c129b74457abb540ff8c84d3b65"
format_stamp: "Formatted at 2026-09-27 12:50:10 on dist-test-slave-lkn4"
server_key: "e513f8a03fed4c7bc8fe8f099fcaa6af"
server_key_iv: "c70960d2b1ec407a2815dbcdcfe82f56"
server_key_version: "encryptionkey@0"
I20260927 12:50:10.090057 18848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:50:10.105531 18848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:50:10.108870 18848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:50:10.109048 18848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:50:10.109627 18848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:50:10.110499 18848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:50:10.110548 18848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:10.110599 18848 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:50:10.110627 18848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:10.119704 18848 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45741
I20260927 12:50:10.119825 18976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45741 every 8 connection(s)
I20260927 12:50:10.120656 18848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:50:10.123667 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 18848
I20260927 12:50:10.123790 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:50:10.124080 392 external_mini_cluster.cc:1428] Setting key cf39d28a15c76651e2d4a523b5e08c85
Sep 27 12:50:10 dist-test-slave-lkn4 krb5kdc[18473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513410, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:50:10.134282 18557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:10.122530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46591 (local address 127.0.98.62:34023)
0927 12:50:10.122649 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:10.122652 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:10.123592 (+ 940us) server_negotiation.cc:407] Connection header received
0927 12:50:10.124597 (+ 1005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:10.124600 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:10.124647 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:10.124722 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:10.126137 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:10.126720 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:10.127834 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:10.128058 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:10.128183 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:10.130514 (+ 2331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:10.130517 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:10.130519 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:50:10.130546 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:10.131971 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:10.132569 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:10.132573 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:10.132574 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:10.132662 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:10.133110 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:10.133113 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:10.133115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:10.133316 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:10.133397 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:10.133985 (+ 588us) server_negotiation.cc:299] Negotiation successful
0927 12:50:10.134142 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 12:50:10.135187 18979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:10.122985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34023 (local address 127.0.98.3:46591)
0927 12:50:10.123450 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:10.123493 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 12:50:10.124387 (+ 894us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:50:10.124917 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:50:10.124926 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:50:10.125360 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:50:10.125954 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:10.125970 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:10.126821 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:10.126824 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:10.127715 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:10.127725 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:10.128174 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:10.128177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:10.128333 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:10.128861 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:50:10.128877 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:50:10.130356 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:50:10.132119 (+ 1763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:50:10.132125 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:50:10.132139 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:50:10.132439 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:50:10.132832 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:50:10.132835 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:50:10.132836 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:50:10.133005 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:50:10.133409 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:50:10.133415 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:50:10.133698 (+ 283us) client_negotiation.cc:769] Sending connection context
0927 12:50:10.133921 (+ 223us) client_negotiation.cc:240] Negotiation successful
0927 12:50:10.134277 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":115,"threads_started":1}
I20260927 12:50:10.136453 18977 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34023
I20260927 12:50:10.136693 18977 heartbeater.cc:464] Registering TS with master...
I20260927 12:50:10.137264 18977 heartbeater.cc:511] Master 127.0.98.62:34023 requested a full tablet report, sending...
I20260927 12:50:10.138252 18518 ts_manager.cc:194] Registered new tserver with Master: 51fa4c129b74457abb540ff8c84d3b65 (127.0.98.3:45741)
I20260927 12:50:10.138783 18518 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46591
I20260927 12:50:10.150094 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 12:50:10.160020 18557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:10.152164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46514 (local address 127.0.98.62:34023)
0927 12:50:10.152331 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:10.152336 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:10.152498 (+ 162us) server_negotiation.cc:407] Connection header received
0927 12:50:10.152632 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:10.152635 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:10.152686 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:10.152753 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:10.153486 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:10.154118 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:10.154918 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:10.155135 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:10.155223 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:10.156776 (+ 1553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:10.156780 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:10.156782 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:50:10.156816 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:10.158292 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:10.158783 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:10.158787 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:10.158788 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:10.158863 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:10.159157 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:10.159160 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:10.159162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:10.159375 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:10.159523 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:10.159775 (+ 252us) server_negotiation.cc:299] Negotiation successful
0927 12:50:10.159864 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 12:50:10.162930 18518 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46514:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:50:10.165983 18518 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:50:10.181444 18988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:10.175333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44675 (local address 127.0.0.1:46336)
0927 12:50:10.175725 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:10.175780 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 12:50:10.175907 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:50:10.177155 (+ 1248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:50:10.177162 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:50:10.177187 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:50:10.177335 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:10.177344 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:10.178476 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:10.178480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:10.179759 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:10.179771 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:10.180811 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:10.180815 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:10.180940 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:10.180986 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 12:50:10.181102 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 12:50:10.181276 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":5,"thread_start_us":125,"threads_started":1}
I20260927 12:50:10.181448 18992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:10.176236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46336 (local address 127.0.98.2:44675)
0927 12:50:10.176645 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:10.176659 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:10.176681 (+ 22us) server_negotiation.cc:407] Connection header received
0927 12:50:10.176757 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:10.176762 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:10.176960 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:10.177145 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:50:10.177475 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:10.178342 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:10.179933 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:10.180693 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:10.180801 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:10.180972 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:10.181148 (+ 176us) server_negotiation.cc:299] Negotiation successful
0927 12:50:10.181269 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":95,"threads_started":1}
I20260927 12:50:10.181887 18989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:10.175913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45741 (local address 127.0.0.1:56082)
0927 12:50:10.176401 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:10.176415 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:50:10.176497 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:50:10.177341 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:50:10.177344 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:50:10.177370 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:50:10.177475 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:10.177481 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:10.178962 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:10.178966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:10.180234 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:10.180244 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:10.181398 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:10.181402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:10.181569 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:10.181617 (+ 48us) client_negotiation.cc:769] Sending connection context
0927 12:50:10.181672 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 12:50:10.181750 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"mutex_wait_us":96,"spinlock_wait_cycles":1920,"thread_start_us":94,"threads_started":1}
I20260927 12:50:10.181988 18990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:10.175494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48772 (local address 127.0.98.1:44439)
0927 12:50:10.176401 (+ 907us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:10.176414 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:10.176436 (+ 22us) server_negotiation.cc:407] Connection header received
0927 12:50:10.176731 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:10.176738 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:10.176970 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:10.177121 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:50:10.177571 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:10.178816 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:10.180261 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:10.181093 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:10.181194 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:10.181335 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:10.181742 (+ 407us) server_negotiation.cc:299] Negotiation successful
0927 12:50:10.181844 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":800,"thread_start_us":85,"threads_started":1}
I20260927 12:50:10.182401 18991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:10.175340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44439 (local address 127.0.0.1:48772)
0927 12:50:10.176139 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:10.176154 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:50:10.176248 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:50:10.177345 (+ 1097us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:50:10.177348 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:50:10.177379 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:50:10.177469 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:10.177475 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:10.178986 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:10.178990 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:10.180146 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:10.180155 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:10.181398 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:10.181402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:10.181553 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:10.181597 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 12:50:10.182186 (+ 589us) client_negotiation.cc:240] Negotiation successful
0927 12:50:10.182271 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":737,"mutex_wait_us":184,"spinlock_wait_cycles":3712,"thread_start_us":98,"threads_started":1}
I20260927 12:50:10.182466 18993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:10.176570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56082 (local address 127.0.98.3:45741)
0927 12:50:10.176960 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:10.176971 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:10.176992 (+ 21us) server_negotiation.cc:407] Connection header received
0927 12:50:10.177058 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:10.177063 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:10.177218 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:10.177345 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:50:10.177588 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:10.178816 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:10.180359 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:10.181094 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:10.181194 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:10.181335 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:10.182224 (+ 889us) server_negotiation.cc:299] Negotiation successful
0927 12:50:10.182330 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":99,"threads_started":1}
I20260927 12:50:10.184808 18777 tablet_service.cc:1467] Processing CreateTablet for tablet 7306a3ed9b0942d2880a234ff36d6ef7 (DEFAULT_TABLE table=test-table [id=50bd14c6db7d463a8b7ff746f597ad7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:50:10.185150 18911 tablet_service.cc:1467] Processing CreateTablet for tablet 7306a3ed9b0942d2880a234ff36d6ef7 (DEFAULT_TABLE table=test-table [id=50bd14c6db7d463a8b7ff746f597ad7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:50:10.185150 18643 tablet_service.cc:1467] Processing CreateTablet for tablet 7306a3ed9b0942d2880a234ff36d6ef7 (DEFAULT_TABLE table=test-table [id=50bd14c6db7d463a8b7ff746f597ad7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:50:10.185855 18777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7306a3ed9b0942d2880a234ff36d6ef7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:50:10.186231 18911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7306a3ed9b0942d2880a234ff36d6ef7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:50:10.186226 18643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7306a3ed9b0942d2880a234ff36d6ef7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:50:10.193254 18994 tablet_bootstrap.cc:492] T 7306a3ed9b0942d2880a234ff36d6ef7 P 095bcd5daf9e4fe6ba7722f0f8853b99: Bootstrap starting.
I20260927 12:50:10.194128 18996 tablet_bootstrap.cc:492] T 7306a3ed9b0942d2880a234ff36d6ef7 P 51fa4c129b74457abb540ff8c84d3b65: Bootstrap starting.
I20260927 12:50:10.194348 18995 tablet_bootstrap.cc:492] T 7306a3ed9b0942d2880a234ff36d6ef7 P 74c6cb864c124739b0dc7e6e8b2cf282: Bootstrap starting.
I20260927 12:50:10.195907 18994 tablet_bootstrap.cc:654] T 7306a3ed9b0942d2880a234ff36d6ef7 P 095bcd5daf9e4fe6ba7722f0f8853b99: Neither blocks nor log segments found. Creating new log.
I20260927 12:50:10.196355 18996 tablet_bootstrap.cc:654] T 7306a3ed9b0942d2880a234ff36d6ef7 P 51fa4c129b74457abb540ff8c84d3b65: Neither blocks nor log segments found. Creating new log.
I20260927 12:50:10.196355 18995 tablet_bootstrap.cc:654] T 7306a3ed9b0942d2880a234ff36d6ef7 P 74c6cb864c124739b0dc7e6e8b2cf282: Neither blocks nor log segments found. Creating new log.
I20260927 12:50:10.196727 18994 log.cc:824] T 7306a3ed9b0942d2880a234ff36d6ef7 P 095bcd5daf9e4fe6ba7722f0f8853b99: Log is configured to *not* fsync() on all Append() calls
I20260927 12:50:10.197193 18995 log.cc:824] T 7306a3ed9b0942d2880a234ff36d6ef7 P 74c6cb864c124739b0dc7e6e8b2cf282: Log is configured to *not* fsync() on all Append() calls
I20260927 12:50:10.197445 18996 log.cc:824] T 7306a3ed9b0942d2880a234ff36d6ef7 P 51fa4c129b74457abb540ff8c84d3b65: Log is configured to *not* fsync() on all Append() calls
I20260927 12:50:10.198911 18994 tablet_bootstrap.cc:492] T 7306a3ed9b0942d2880a234ff36d6ef7 P 095bcd5daf9e4fe6ba7722f0f8853b99: No bootstrap required, opened a new log
I20260927 12:50:10.199117 18994 ts_tablet_manager.cc:1397] T 7306a3ed9b0942d2880a234ff36d6ef7 P 095bcd5daf9e4fe6ba7722f0f8853b99: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 12:50:10.199286 18996 tablet_bootstrap.cc:492] T 7306a3ed9b0942d2880a234ff36d6ef7 P 51fa4c129b74457abb540ff8c84d3b65: No bootstrap required, opened a new log
I20260927 12:50:10.199460 18996 ts_tablet_manager.cc:1397] T 7306a3ed9b0942d2880a234ff36d6ef7 P 51fa4c129b74457abb540ff8c84d3b65: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 12:50:10.202188 18994 raft_consensus.cc:348] T 7306a3ed9b0942d2880a234ff36d6ef7 P 095bcd5daf9e4fe6ba7722f0f8853b99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c6cb864c124739b0dc7e6e8b2cf282" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44439 } } peers { permanent_uuid: "095bcd5daf9e4fe6ba7722f0f8853b99" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44675 } } peers { permanent_uuid: "51fa4c129b74457abb540ff8c84d3b65" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45741 } }
I20260927 12:50:10.202195 18996 raft_consensus.cc:348] T 7306a3ed9b0942d2880a234ff36d6ef7 P 51fa4c129b74457abb540ff8c84d3b65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c6cb864c124739b0dc7e6e8b2cf282" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44439 } } peers { permanent_uuid: "095bcd5daf9e4fe6ba7722f0f8853b99" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44675 } } peers { permanent_uuid: "51fa4c129b74457abb540ff8c84d3b65" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45741 } }
I20260927 12:50:10.202458 18994 raft_consensus.cc:374] T 7306a3ed9b0942d2880a234ff36d6ef7 P 095bcd5daf9e4fe6ba7722f0f8853b99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:50:10.202515 18996 raft_consensus.cc:374] T 7306a3ed9b0942d2880a234ff36d6ef7 P 51fa4c129b74457abb540ff8c84d3b65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:50:10.202519 18994 raft_consensus.cc:729] T 7306a3ed9b0942d2880a234ff36d6ef7 P 095bcd5daf9e4fe6ba7722f0f8853b99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 095bcd5daf9e4fe6ba7722f0f8853b99, State: Initialized, Role: FOLLOWER
I20260927 12:50:10.202560 18996 raft_consensus.cc:729] T 7306a3ed9b0942d2880a234ff36d6ef7 P 51fa4c129b74457abb540ff8c84d3b65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51fa4c129b74457abb540ff8c84d3b65, State: Initialized, Role: FOLLOWER
I20260927 12:50:10.202979 18994 consensus_queue.cc:260] T 7306a3ed9b0942d2880a234ff36d6ef7 P 095bcd5daf9e4fe6ba7722f0f8853b99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c6cb864c124739b0dc7e6e8b2cf282" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44439 } } peers { permanent_uuid: "095bcd5daf9e4fe6ba7722f0f8853b99" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44675 } } peers { permanent_uuid: "51fa4c129b74457abb540ff8c84d3b65" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45741 } }
I20260927 12:50:10.202983 18996 consensus_queue.cc:260] T 7306a3ed9b0942d2880a234ff36d6ef7 P 51fa4c129b74457abb540ff8c84d3b65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c6cb864c124739b0dc7e6e8b2cf282" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44439 } } peers { permanent_uuid: "095bcd5daf9e4fe6ba7722f0f8853b99" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44675 } } peers { permanent_uuid: "51fa4c129b74457abb540ff8c84d3b65" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45741 } }
I20260927 12:50:10.203639 18995 tablet_bootstrap.cc:492] T 7306a3ed9b0942d2880a234ff36d6ef7 P 74c6cb864c124739b0dc7e6e8b2cf282: No bootstrap required, opened a new log
I20260927 12:50:10.203846 18995 ts_tablet_manager.cc:1397] T 7306a3ed9b0942d2880a234ff36d6ef7 P 74c6cb864c124739b0dc7e6e8b2cf282: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260927 12:50:10.203924 18977 heartbeater.cc:503] Master 127.0.98.62:34023 was elected leader, sending a full tablet report...
I20260927 12:50:10.203994 18996 ts_tablet_manager.cc:1428] T 7306a3ed9b0942d2880a234ff36d6ef7 P 51fa4c129b74457abb540ff8c84d3b65: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260927 12:50:10.204001 18994 ts_tablet_manager.cc:1428] T 7306a3ed9b0942d2880a234ff36d6ef7 P 095bcd5daf9e4fe6ba7722f0f8853b99: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 12:50:10.204246 18843 heartbeater.cc:503] Master 127.0.98.62:34023 was elected leader, sending a full tablet report...
I20260927 12:50:10.208024 18995 raft_consensus.cc:348] T 7306a3ed9b0942d2880a234ff36d6ef7 P 74c6cb864c124739b0dc7e6e8b2cf282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c6cb864c124739b0dc7e6e8b2cf282" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44439 } } peers { permanent_uuid: "095bcd5daf9e4fe6ba7722f0f8853b99" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44675 } } peers { permanent_uuid: "51fa4c129b74457abb540ff8c84d3b65" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45741 } }
I20260927 12:50:10.208307 18995 raft_consensus.cc:374] T 7306a3ed9b0942d2880a234ff36d6ef7 P 74c6cb864c124739b0dc7e6e8b2cf282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:50:10.208370 18995 raft_consensus.cc:729] T 7306a3ed9b0942d2880a234ff36d6ef7 P 74c6cb864c124739b0dc7e6e8b2cf282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74c6cb864c124739b0dc7e6e8b2cf282, State: Initialized, Role: FOLLOWER
I20260927 12:50:10.208823 18995 consensus_queue.cc:260] T 7306a3ed9b0942d2880a234ff36d6ef7 P 74c6cb864c124739b0dc7e6e8b2cf282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c6cb864c124739b0dc7e6e8b2cf282" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44439 } } peers { permanent_uuid: "095bcd5daf9e4fe6ba7722f0f8853b99" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44675 } } peers { permanent_uuid: "51fa4c129b74457abb540ff8c84d3b65" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45741 } }
I20260927 12:50:10.209498 18709 heartbeater.cc:503] Master 127.0.98.62:34023 was elected leader, sending a full tablet report...
I20260927 12:50:10.209829 18995 ts_tablet_manager.cc:1428] T 7306a3ed9b0942d2880a234ff36d6ef7 P 74c6cb864c124739b0dc7e6e8b2cf282: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 12:50:10.210065 19000 raft_consensus.cc:482] T 7306a3ed9b0942d2880a234ff36d6ef7 P 095bcd5daf9e4fe6ba7722f0f8853b99 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:50:10.210246 19000 raft_consensus.cc:504] T 7306a3ed9b0942d2880a234ff36d6ef7 P 095bcd5daf9e4fe6ba7722f0f8853b99 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c6cb864c124739b0dc7e6e8b2cf282" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44439 } } peers { permanent_uuid: "095bcd5daf9e4fe6ba7722f0f8853b99" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44675 } } peers { permanent_uuid: "51fa4c129b74457abb540ff8c84d3b65" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45741 } }
I20260927 12:50:10.211390 19000 leader_election.cc:288] T 7306a3ed9b0942d2880a234ff36d6ef7 P 095bcd5daf9e4fe6ba7722f0f8853b99 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74c6cb864c124739b0dc7e6e8b2cf282 (127.0.98.1:44439), 51fa4c129b74457abb540ff8c84d3b65 (127.0.98.3:45741)
I20260927 12:50:10.215737 18990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:10.211985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45997 (local address 127.0.98.1:44439)
0927 12:50:10.212091 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:10.212096 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:10.212108 (+ 12us) server_negotiation.cc:407] Connection header received
0927 12:50:10.212149 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:10.212153 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:10.212194 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:10.212265 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:50:10.212628 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:10.213314 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:10.214795 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:10.215211 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:10.215306 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:10.215367 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:10.215533 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 12:50:10.215603 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 12:50:10.215788 19003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:10.211545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44439 (local address 127.0.98.2:45997)
0927 12:50:10.211907 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:10.211922 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:50:10.212020 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:50:10.212317 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:50:10.212319 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:50:10.212336 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:50:10.212484 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:10.212491 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:10.213414 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:10.213417 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:10.214648 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:10.214659 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:10.215320 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:10.215324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:10.215427 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:10.215448 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 12:50:10.215511 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 12:50:10.215579 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":107}
I20260927 12:50:10.215803 18845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:10.211602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45741 (local address 127.0.98.2:49229)
0927 12:50:10.212028 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:10.212043 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:50:10.212124 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:50:10.212668 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:50:10.212670 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:50:10.212694 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:50:10.212788 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:10.212793 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:10.213509 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:10.213511 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:10.214623 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:10.214634 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:10.215280 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:10.215284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:10.215400 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:10.215423 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 12:50:10.215491 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 12:50:10.215573 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":39,"thread_start_us":148,"threads_started":1}
I20260927 12:50:10.215873 18993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:10.212308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49229 (local address 127.0.98.3:45741)
0927 12:50:10.212474 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:10.212479 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:10.212492 (+ 13us) server_negotiation.cc:407] Connection header received
0927 12:50:10.212534 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:10.212537 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:10.212596 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:10.212661 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:50:10.212886 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:10.213410 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:10.214732 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:10.215167 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:10.215261 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:10.215323 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:10.215693 (+ 370us) server_negotiation.cc:299] Negotiation successful
0927 12:50:10.215772 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 12:50:10.216324 18663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7306a3ed9b0942d2880a234ff36d6ef7" candidate_uuid: "095bcd5daf9e4fe6ba7722f0f8853b99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74c6cb864c124739b0dc7e6e8b2cf282" is_pre_election: true
I20260927 12:50:10.216645 18663 raft_consensus.cc:2454] T 7306a3ed9b0942d2880a234ff36d6ef7 P 74c6cb864c124739b0dc7e6e8b2cf282 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 095bcd5daf9e4fe6ba7722f0f8853b99 in term 0.
I20260927 12:50:10.216656 18931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7306a3ed9b0942d2880a234ff36d6ef7" candidate_uuid: "095bcd5daf9e4fe6ba7722f0f8853b99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51fa4c129b74457abb540ff8c84d3b65" is_pre_election: true
I20260927 12:50:10.216912 18931 raft_consensus.cc:2454] T 7306a3ed9b0942d2880a234ff36d6ef7 P 51fa4c129b74457abb540ff8c84d3b65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 095bcd5daf9e4fe6ba7722f0f8853b99 in term 0.
I20260927 12:50:10.217095 18732 leader_election.cc:302] T 7306a3ed9b0942d2880a234ff36d6ef7 P 095bcd5daf9e4fe6ba7722f0f8853b99 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 095bcd5daf9e4fe6ba7722f0f8853b99, 74c6cb864c124739b0dc7e6e8b2cf282; no voters:
I20260927 12:50:10.217415 19000 raft_consensus.cc:2781] T 7306a3ed9b0942d2880a234ff36d6ef7 P 095bcd5daf9e4fe6ba7722f0f8853b99 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:50:10.217492 19000 raft_consensus.cc:482] T 7306a3ed9b0942d2880a234ff36d6ef7 P 095bcd5daf9e4fe6ba7722f0f8853b99 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:50:10.217541 19000 raft_consensus.cc:3037] T 7306a3ed9b0942d2880a234ff36d6ef7 P 095bcd5daf9e4fe6ba7722f0f8853b99 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:50:10.218679 19000 raft_consensus.cc:504] T 7306a3ed9b0942d2880a234ff36d6ef7 P 095bcd5daf9e4fe6ba7722f0f8853b99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c6cb864c124739b0dc7e6e8b2cf282" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44439 } } peers { permanent_uuid: "095bcd5daf9e4fe6ba7722f0f8853b99" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44675 } } peers { permanent_uuid: "51fa4c129b74457abb540ff8c84d3b65" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45741 } }
I20260927 12:50:10.218992 19000 leader_election.cc:288] T 7306a3ed9b0942d2880a234ff36d6ef7 P 095bcd5daf9e4fe6ba7722f0f8853b99 [CANDIDATE]: Term 1 election: Requested vote from peers 74c6cb864c124739b0dc7e6e8b2cf282 (127.0.98.1:44439), 51fa4c129b74457abb540ff8c84d3b65 (127.0.98.3:45741)
I20260927 12:50:10.219323 18931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7306a3ed9b0942d2880a234ff36d6ef7" candidate_uuid: "095bcd5daf9e4fe6ba7722f0f8853b99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51fa4c129b74457abb540ff8c84d3b65"
I20260927 12:50:10.219435 18931 raft_consensus.cc:3037] T 7306a3ed9b0942d2880a234ff36d6ef7 P 51fa4c129b74457abb540ff8c84d3b65 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:50:10.220434 18931 raft_consensus.cc:2454] T 7306a3ed9b0942d2880a234ff36d6ef7 P 51fa4c129b74457abb540ff8c84d3b65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 095bcd5daf9e4fe6ba7722f0f8853b99 in term 1.
I20260927 12:50:10.220680 18663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7306a3ed9b0942d2880a234ff36d6ef7" candidate_uuid: "095bcd5daf9e4fe6ba7722f0f8853b99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74c6cb864c124739b0dc7e6e8b2cf282"
I20260927 12:50:10.220745 18730 leader_election.cc:302] T 7306a3ed9b0942d2880a234ff36d6ef7 P 095bcd5daf9e4fe6ba7722f0f8853b99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 095bcd5daf9e4fe6ba7722f0f8853b99, 51fa4c129b74457abb540ff8c84d3b65; no voters:
I20260927 12:50:10.220796 18663 raft_consensus.cc:3037] T 7306a3ed9b0942d2880a234ff36d6ef7 P 74c6cb864c124739b0dc7e6e8b2cf282 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:50:10.220896 19000 raft_consensus.cc:2781] T 7306a3ed9b0942d2880a234ff36d6ef7 P 095bcd5daf9e4fe6ba7722f0f8853b99 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:50:10.221123 19000 raft_consensus.cc:686] T 7306a3ed9b0942d2880a234ff36d6ef7 P 095bcd5daf9e4fe6ba7722f0f8853b99 [term 1 LEADER]: Becoming Leader. State: Replica: 095bcd5daf9e4fe6ba7722f0f8853b99, State: Running, Role: LEADER
I20260927 12:50:10.221485 19000 consensus_queue.cc:237] T 7306a3ed9b0942d2880a234ff36d6ef7 P 095bcd5daf9e4fe6ba7722f0f8853b99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c6cb864c124739b0dc7e6e8b2cf282" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44439 } } peers { permanent_uuid: "095bcd5daf9e4fe6ba7722f0f8853b99" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44675 } } peers { permanent_uuid: "51fa4c129b74457abb540ff8c84d3b65" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45741 } }
I20260927 12:50:10.222116 18663 raft_consensus.cc:2454] T 7306a3ed9b0942d2880a234ff36d6ef7 P 74c6cb864c124739b0dc7e6e8b2cf282 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 095bcd5daf9e4fe6ba7722f0f8853b99 in term 1.
I20260927 12:50:10.225075 18518 catalog_manager.cc:5495] T 7306a3ed9b0942d2880a234ff36d6ef7 P 095bcd5daf9e4fe6ba7722f0f8853b99 reported cstate change: term changed from 0 to 1, leader changed from <none> to 095bcd5daf9e4fe6ba7722f0f8853b99 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "095bcd5daf9e4fe6ba7722f0f8853b99" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74c6cb864c124739b0dc7e6e8b2cf282" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "095bcd5daf9e4fe6ba7722f0f8853b99" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51fa4c129b74457abb540ff8c84d3b65" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45741 } health_report { overall_health: UNKNOWN } } }
I20260927 12:50:10.246742 18992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:10.242965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46340 (local address 127.0.98.2:44675)
0927 12:50:10.243093 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:10.243097 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:10.243111 (+ 14us) server_negotiation.cc:407] Connection header received
0927 12:50:10.243237 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:10.243240 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:10.243294 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:10.243381 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:50:10.243755 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:10.244277 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:10.245133 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:10.245358 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:10.245435 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:10.245838 (+ 403us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:50:10.246339 (+ 501us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:50:10.246451 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:10.246562 (+ 111us) server_negotiation.cc:299] Negotiation successful
0927 12:50:10.246598 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 12:50:10.252990 18931 raft_consensus.cc:1261] T 7306a3ed9b0942d2880a234ff36d6ef7 P 51fa4c129b74457abb540ff8c84d3b65 [term 1 FOLLOWER]: Refusing update from remote peer 095bcd5daf9e4fe6ba7722f0f8853b99: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:50:10.252990 18663 raft_consensus.cc:1261] T 7306a3ed9b0942d2880a234ff36d6ef7 P 74c6cb864c124739b0dc7e6e8b2cf282 [term 1 FOLLOWER]: Refusing update from remote peer 095bcd5daf9e4fe6ba7722f0f8853b99: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:50:10.253679 19000 consensus_queue.cc:1035] T 7306a3ed9b0942d2880a234ff36d6ef7 P 095bcd5daf9e4fe6ba7722f0f8853b99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51fa4c129b74457abb540ff8c84d3b65" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:50:10.253876 19004 consensus_queue.cc:1035] T 7306a3ed9b0942d2880a234ff36d6ef7 P 095bcd5daf9e4fe6ba7722f0f8853b99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74c6cb864c124739b0dc7e6e8b2cf282" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:50:10.261875 19008 mvcc.cc:205] Tried to move back new op lower bound from 7333942928388644864 to 7333942928270503936. Current Snapshot: MvccSnapshot[applied={T|T < 7333942928388644864}]
I20260927 12:50:10.261933 19009 mvcc.cc:205] Tried to move back new op lower bound from 7333942928388644864 to 7333942928270503936. Current Snapshot: MvccSnapshot[applied={T|T < 7333942928388644864}]
I20260927 12:50:10.270027 18517 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:50:10.270221 18517 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:50:10.272596 18517 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e0fc18da404e4742901121732eeb27a0: Sending DeleteTablet for 3 replicas of tablet 7306a3ed9b0942d2880a234ff36d6ef7
I20260927 12:50:10.273428 18643 tablet_service.cc:1514] Processing DeleteTablet for tablet 7306a3ed9b0942d2880a234ff36d6ef7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:50:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48772
I20260927 12:50:10.273484 18911 tablet_service.cc:1514] Processing DeleteTablet for tablet 7306a3ed9b0942d2880a234ff36d6ef7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:50:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56082
I20260927 12:50:10.274024 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 18580
I20260927 12:50:10.274200 18777 tablet_service.cc:1514] Processing DeleteTablet for tablet 7306a3ed9b0942d2880a234ff36d6ef7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:50:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46336
I20260927 12:50:10.275081 19019 tablet_replica.cc:330] T 7306a3ed9b0942d2880a234ff36d6ef7 P 51fa4c129b74457abb540ff8c84d3b65: stopping tablet replica
I20260927 12:50:10.275290 19020 tablet_replica.cc:330] T 7306a3ed9b0942d2880a234ff36d6ef7 P 095bcd5daf9e4fe6ba7722f0f8853b99: stopping tablet replica
I20260927 12:50:10.275408 19019 raft_consensus.cc:2229] T 7306a3ed9b0942d2880a234ff36d6ef7 P 51fa4c129b74457abb540ff8c84d3b65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:50:10.275691 19020 raft_consensus.cc:2229] T 7306a3ed9b0942d2880a234ff36d6ef7 P 095bcd5daf9e4fe6ba7722f0f8853b99 [term 1 LEADER]: Raft consensus shutting down.
I20260927 12:50:10.275741 19019 raft_consensus.cc:2258] T 7306a3ed9b0942d2880a234ff36d6ef7 P 51fa4c129b74457abb540ff8c84d3b65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:50:10.276155 19020 raft_consensus.cc:2258] T 7306a3ed9b0942d2880a234ff36d6ef7 P 095bcd5daf9e4fe6ba7722f0f8853b99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:50:10.277302 19019 ts_tablet_manager.cc:1905] T 7306a3ed9b0942d2880a234ff36d6ef7 P 51fa4c129b74457abb540ff8c84d3b65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:50:10.277679 19020 ts_tablet_manager.cc:1905] T 7306a3ed9b0942d2880a234ff36d6ef7 P 095bcd5daf9e4fe6ba7722f0f8853b99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:50:10.279589 19019 ts_tablet_manager.cc:1918] T 7306a3ed9b0942d2880a234ff36d6ef7 P 51fa4c129b74457abb540ff8c84d3b65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:50:10.279677 19019 log.cc:1196] T 7306a3ed9b0942d2880a234ff36d6ef7 P 51fa4c129b74457abb540ff8c84d3b65: Deleting WAL directory at /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/minicluster-data/ts-2/wal/wals/7306a3ed9b0942d2880a234ff36d6ef7
I20260927 12:50:10.279953 19019 ts_tablet_manager.cc:1939] T 7306a3ed9b0942d2880a234ff36d6ef7 P 51fa4c129b74457abb540ff8c84d3b65: Deleting consensus metadata
I20260927 12:50:10.280260 19020 ts_tablet_manager.cc:1918] T 7306a3ed9b0942d2880a234ff36d6ef7 P 095bcd5daf9e4fe6ba7722f0f8853b99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:50:10.280352 19020 log.cc:1196] T 7306a3ed9b0942d2880a234ff36d6ef7 P 095bcd5daf9e4fe6ba7722f0f8853b99: Deleting WAL directory at /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790513161160093-392-0/minicluster-data/ts-1/wal/wals/7306a3ed9b0942d2880a234ff36d6ef7
I20260927 12:50:10.280640 19020 ts_tablet_manager.cc:1939] T 7306a3ed9b0942d2880a234ff36d6ef7 P 095bcd5daf9e4fe6ba7722f0f8853b99: Deleting consensus metadata
I20260927 12:50:10.281126 18502 catalog_manager.cc:4841] TS 51fa4c129b74457abb540ff8c84d3b65 (127.0.98.3:45741): tablet 7306a3ed9b0942d2880a234ff36d6ef7 (table test-table [id=50bd14c6db7d463a8b7ff746f597ad7b]) successfully deleted
I20260927 12:50:10.281679 18503 catalog_manager.cc:4841] TS 095bcd5daf9e4fe6ba7722f0f8853b99 (127.0.98.2:44675): tablet 7306a3ed9b0942d2880a234ff36d6ef7 (table test-table [id=50bd14c6db7d463a8b7ff746f597ad7b]) successfully deleted
W20260927 12:50:10.284761 18504 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44439 (error 108)
I20260927 12:50:10.285588 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 18714
I20260927 12:50:10.285858 18989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:10.285543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44439 (local address 127.0.0.1:48788)
0927 12:50:10.285689 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:10.285770 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 12:50:10.287257 18504 catalog_manager.cc:4584] TS 74c6cb864c124739b0dc7e6e8b2cf282 (127.0.98.1:44439): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7306a3ed9b0942d2880a234ff36d6ef7: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44439: connect: Connection refused (error 111)
I20260927 12:50:10.291239 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 18848
I20260927 12:50:10.296443 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 18486
2026-09-27T12:50:10Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2166 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-27T12:50:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:50:10Z Disabled control of system clock
I20260927 12:50:10.324777 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790513161160093-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790513161160093-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790513161160093-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42299
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42299
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 12:50:10.437058 19025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:50:10.437353 19025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:50:10.437390 19025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:50:10.441675 19025 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:50:10.441752 19025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:50:10.441766 19025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:50:10.441781 19025 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:50:10.441795 19025 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:50:10.446797 19025 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790513161160093-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790513161160093-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42299
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790513161160093-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:50:10.447968 19025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:50:10.449647 19025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 12:50:10.458550 19025 server_base.cc:1034] running on GCE node
W20260927 12:50:10.458470 19031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:50:10.458441 19030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:50:10.458463 19033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:50:10.459471 19025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:50:10.460685 19025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:50:10.461853 19025 hybrid_clock.cc:648] HybridClock initialized: now 1790513410461832 us; error 39 us; skew 500 ppm
I20260927 12:50:10.463856 19025 webserver.cc:460] Webserver started at http://127.0.98.62:46275/ using document root <none> and password file <none>
I20260927 12:50:10.464470 19025 fs_manager.cc:359] Metadata directory not provided
I20260927 12:50:10.464542 19025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:50:10.464744 19025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:50:10.466964 19025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790513161160093-392-0/minicluster-data/master-0/data/instance:
uuid: "75b2ac1e75cf46d68cb6ab0af3797758"
format_stamp: "Formatted at 2026-09-27 12:50:10 on dist-test-slave-lkn4"
server_key: "614f6f17d8a20ca3b59bc1aa66f37877"
server_key_iv: "a2b67b69dce90530ec769c4c58434d21"
server_key_version: "encryptionkey@0"
I20260927 12:50:10.467536 19025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790513161160093-392-0/minicluster-data/master-0/wal/instance:
uuid: "75b2ac1e75cf46d68cb6ab0af3797758"
format_stamp: "Formatted at 2026-09-27 12:50:10 on dist-test-slave-lkn4"
server_key: "614f6f17d8a20ca3b59bc1aa66f37877"
server_key_iv: "a2b67b69dce90530ec769c4c58434d21"
server_key_version: "encryptionkey@0"
I20260927 12:50:10.471535 19025 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 12:50:10.474411 19039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:10.475804 19025 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 12:50:10.475929 19025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790513161160093-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790513161160093-392-0/minicluster-data/master-0/wal
uuid: "75b2ac1e75cf46d68cb6ab0af3797758"
format_stamp: "Formatted at 2026-09-27 12:50:10 on dist-test-slave-lkn4"
server_key: "614f6f17d8a20ca3b59bc1aa66f37877"
server_key_iv: "a2b67b69dce90530ec769c4c58434d21"
server_key_version: "encryptionkey@0"
I20260927 12:50:10.476063 19025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790513161160093-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790513161160093-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790513161160093-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:50:10.494953 19025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:50:10.495679 19025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:50:10.495864 19025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:50:10.502965 19025 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42299
I20260927 12:50:10.502996 19091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42299 every 8 connection(s)
I20260927 12:50:10.503966 19025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:50:10.507360 19092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:50:10.510946 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 19025
I20260927 12:50:10.511055 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790513161160093-392-0/minicluster-data/master-0/wal/instance
I20260927 12:50:10.511322 392 external_mini_cluster.cc:1428] Setting key 4b65453df28826899fb1eb804cd9525d
I20260927 12:50:10.513279 19092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75b2ac1e75cf46d68cb6ab0af3797758: Bootstrap starting.
I20260927 12:50:10.516044 19092 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75b2ac1e75cf46d68cb6ab0af3797758: Neither blocks nor log segments found. Creating new log.
I20260927 12:50:10.516940 19092 log.cc:824] T 00000000000000000000000000000000 P 75b2ac1e75cf46d68cb6ab0af3797758: Log is configured to *not* fsync() on all Append() calls
I20260927 12:50:10.519028 19092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75b2ac1e75cf46d68cb6ab0af3797758: No bootstrap required, opened a new log
I20260927 12:50:10.521548 19095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:10.512861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37386 (local address 127.0.98.62:42299)
0927 12:50:10.513585 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:10.513604 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:10.513656 (+ 52us) server_negotiation.cc:407] Connection header received
0927 12:50:10.514522 (+ 866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:10.514560 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:10.515085 (+ 525us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:10.515442 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:10.515897 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:10.517364 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:10.518290 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:10.518767 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:10.518911 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:10.519716 (+ 805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:10.519721 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:10.519735 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:50:10.520110 (+ 375us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:10.520165 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:50:10.520184 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:10.520268 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:10.520397 (+ 129us) server_negotiation.cc:299] Negotiation successful
0927 12:50:10.520666 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":117,"threads_started":1}
I20260927 12:50:10.522222 19092 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75b2ac1e75cf46d68cb6ab0af3797758 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75b2ac1e75cf46d68cb6ab0af3797758" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42299 } }
I20260927 12:50:10.522450 19092 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75b2ac1e75cf46d68cb6ab0af3797758 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:50:10.523466 19092 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75b2ac1e75cf46d68cb6ab0af3797758 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75b2ac1e75cf46d68cb6ab0af3797758, State: Initialized, Role: FOLLOWER
I20260927 12:50:10.523944 19092 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75b2ac1e75cf46d68cb6ab0af3797758 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75b2ac1e75cf46d68cb6ab0af3797758" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42299 } }
I20260927 12:50:10.524096 19092 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75b2ac1e75cf46d68cb6ab0af3797758 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:50:10.524156 19092 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75b2ac1e75cf46d68cb6ab0af3797758 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:50:10.524262 19092 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75b2ac1e75cf46d68cb6ab0af3797758 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:50:10.525306 19092 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75b2ac1e75cf46d68cb6ab0af3797758 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75b2ac1e75cf46d68cb6ab0af3797758" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42299 } }
I20260927 12:50:10.525691 19092 leader_election.cc:302] T 00000000000000000000000000000000 P 75b2ac1e75cf46d68cb6ab0af3797758 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75b2ac1e75cf46d68cb6ab0af3797758; no voters:
I20260927 12:50:10.526000 19092 leader_election.cc:288] T 00000000000000000000000000000000 P 75b2ac1e75cf46d68cb6ab0af3797758 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:50:10.526132 19097 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75b2ac1e75cf46d68cb6ab0af3797758 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:50:10.526454 19097 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75b2ac1e75cf46d68cb6ab0af3797758 [term 1 LEADER]: Becoming Leader. State: Replica: 75b2ac1e75cf46d68cb6ab0af3797758, State: Running, Role: LEADER
I20260927 12:50:10.526816 19097 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75b2ac1e75cf46d68cb6ab0af3797758 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75b2ac1e75cf46d68cb6ab0af3797758" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42299 } }
I20260927 12:50:10.527263 19092 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75b2ac1e75cf46d68cb6ab0af3797758 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:50:10.528358 19099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75b2ac1e75cf46d68cb6ab0af3797758 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75b2ac1e75cf46d68cb6ab0af3797758. Latest consensus state: current_term: 1 leader_uuid: "75b2ac1e75cf46d68cb6ab0af3797758" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75b2ac1e75cf46d68cb6ab0af3797758" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42299 } } }
I20260927 12:50:10.528488 19099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75b2ac1e75cf46d68cb6ab0af3797758 [sys.catalog]: This master's current role is: LEADER
I20260927 12:50:10.528781 19098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75b2ac1e75cf46d68cb6ab0af3797758 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75b2ac1e75cf46d68cb6ab0af3797758" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75b2ac1e75cf46d68cb6ab0af3797758" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42299 } } }
I20260927 12:50:10.528880 19098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75b2ac1e75cf46d68cb6ab0af3797758 [sys.catalog]: This master's current role is: LEADER
I20260927 12:50:10.528863 19106 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:50:10.531654 19106 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:50:10.536862 19106 catalog_manager.cc:1347] Generated new cluster ID: 7ed658caa986499caf80a82cc757b901
I20260927 12:50:10.536949 19106 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:50:10.553663 19106 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:50:10.554725 19106 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:50:10.563899 19106 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75b2ac1e75cf46d68cb6ab0af3797758: Generated new TSK 0
I20260927 12:50:10.564807 19106 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:50:10.579015 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790513161160093-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790513161160093-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790513161160093-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42299
--builtin_ntp_servers=127.0.98.20:33297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 12:50:10.692687 19116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:50:10.692979 19116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:50:10.693019 19116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:50:10.697314 19116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:50:10.697417 19116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:50:10.697491 19116 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:50:10.702582 19116 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790513161160093-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790513161160093-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42299
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790513161160093-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:50:10.703771 19116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:50:10.705199 19116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:50:10.715348 19122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:50:10.715396 19116 server_base.cc:1034] running on GCE node
W20260927 12:50:10.715401 19121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:50:10.715348 19124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:50:10.716079 19116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:50:10.716774 19116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:50:10.717928 19116 hybrid_clock.cc:648] HybridClock initialized: now 1790513410717907 us; error 33 us; skew 500 ppm
I20260927 12:50:10.719815 19116 webserver.cc:460] Webserver started at http://127.0.98.1:40793/ using document root <none> and password file <none>
I20260927 12:50:10.720381 19116 fs_manager.cc:359] Metadata directory not provided
I20260927 12:50:10.720443 19116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:50:10.720641 19116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:50:10.722756 19116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790513161160093-392-0/minicluster-data/ts-0/data/instance:
uuid: "29daecf51a094db79140803abd92f8aa"
format_stamp: "Formatted at 2026-09-27 12:50:10 on dist-test-slave-lkn4"
server_key: "9be7e59b7776e386965c8779c3ff8cef"
server_key_iv: "05b6d516bd9ab591b8394cc672ef5f2a"
server_key_version: "encryptionkey@0"
I20260927 12:50:10.723284 19116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790513161160093-392-0/minicluster-data/ts-0/wal/instance:
uuid: "29daecf51a094db79140803abd92f8aa"
format_stamp: "Formatted at 2026-09-27 12:50:10 on dist-test-slave-lkn4"
server_key: "9be7e59b7776e386965c8779c3ff8cef"
server_key_iv: "05b6d516bd9ab591b8394cc672ef5f2a"
server_key_version: "encryptionkey@0"
I20260927 12:50:10.727254 19116 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 12:50:10.729717 19130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:10.731030 19116 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:50:10.731133 19116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790513161160093-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790513161160093-392-0/minicluster-data/ts-0/wal
uuid: "29daecf51a094db79140803abd92f8aa"
format_stamp: "Formatted at 2026-09-27 12:50:10 on dist-test-slave-lkn4"
server_key: "9be7e59b7776e386965c8779c3ff8cef"
server_key_iv: "05b6d516bd9ab591b8394cc672ef5f2a"
server_key_version: "encryptionkey@0"
I20260927 12:50:10.731238 19116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790513161160093-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790513161160093-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790513161160093-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:50:10.754232 19116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:50:10.754982 19116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:50:10.755162 19116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:50:10.755882 19116 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:50:10.757005 19116 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:50:10.757058 19116 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:10.757103 19116 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:50:10.757133 19116 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:10.766014 19116 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33377
I20260927 12:50:10.766064 19243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33377 every 8 connection(s)
I20260927 12:50:10.767021 19116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:50:10.776175 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 19116
I20260927 12:50:10.776324 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790513161160093-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:50:10.776655 392 external_mini_cluster.cc:1428] Setting key b1cdcfb15d5cc9acbc76ad53e9d5a6c5
I20260927 12:50:10.776620 19095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:10.768874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59509 (local address 127.0.98.62:42299)
0927 12:50:10.769016 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:10.769021 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:10.770259 (+ 1238us) server_negotiation.cc:407] Connection header received
0927 12:50:10.771622 (+ 1363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:10.771624 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:10.771671 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:10.771734 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:10.772774 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:10.773296 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:10.774534 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:10.774742 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:10.774811 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:10.776080 (+ 1269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:10.776084 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:10.776086 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:50:10.776102 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:10.776127 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:50:10.776138 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:10.776223 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:10.776390 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 12:50:10.776454 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 12:50:10.777575 19246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:10.769364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42299 (local address 127.0.98.1:59509)
0927 12:50:10.770109 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:10.770156 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 12:50:10.771418 (+ 1262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:50:10.771953 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:50:10.771961 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:50:10.772052 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:50:10.772635 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:10.772658 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:10.773411 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:10.773414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:10.774433 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:10.774443 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:10.774885 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:10.774888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:10.775049 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:10.775787 (+ 738us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 12:50:10.775789 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:50:10.775834 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 12:50:10.775902 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:50:10.776235 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:50:10.776242 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:50:10.776247 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 12:50:10.776377 (+ 130us) client_negotiation.cc:240] Negotiation successful
0927 12:50:10.776640 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":125,"threads_started":1}
I20260927 12:50:10.778708 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790513161160093-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790513161160093-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790513161160093-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42299
--builtin_ntp_servers=127.0.98.20:33297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 12:50:10.778831 19244 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42299
I20260927 12:50:10.779067 19244 heartbeater.cc:464] Registering TS with master...
I20260927 12:50:10.779645 19244 heartbeater.cc:511] Master 127.0.98.62:42299 requested a full tablet report, sending...
I20260927 12:50:10.781352 19056 ts_manager.cc:194] Registered new tserver with Master: 29daecf51a094db79140803abd92f8aa (127.0.98.1:33377)
I20260927 12:50:10.782851 19056 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:59509
W20260927 12:50:10.889362 19247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:50:10.889634 19247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:50:10.889672 19247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:50:10.893714 19247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:50:10.893787 19247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:50:10.893846 19247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:50:10.898756 19247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790513161160093-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790513161160093-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42299
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790513161160093-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:50:10.899868 19247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:50:10.901221 19247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 12:50:10.910867 19247 server_base.cc:1034] running on GCE node
W20260927 12:50:10.910976 19252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:50:10.910836 19255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:50:10.910835 19253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:50:10.911664 19247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:50:10.912266 19247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:50:10.913441 19247 hybrid_clock.cc:648] HybridClock initialized: now 1790513410913418 us; error 33 us; skew 500 ppm
I20260927 12:50:10.915326 19247 webserver.cc:460] Webserver started at http://127.0.98.2:43281/ using document root <none> and password file <none>
I20260927 12:50:10.915977 19247 fs_manager.cc:359] Metadata directory not provided
I20260927 12:50:10.916045 19247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:50:10.916219 19247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:50:10.918358 19247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790513161160093-392-0/minicluster-data/ts-1/data/instance:
uuid: "e4422a27f949458fbd6184e6585d7d9b"
format_stamp: "Formatted at 2026-09-27 12:50:10 on dist-test-slave-lkn4"
server_key: "045193871f5756c8a1e96372b3890c65"
server_key_iv: "58c0ddcdb8feea7936af4e6a74c49a58"
server_key_version: "encryptionkey@0"
I20260927 12:50:10.918901 19247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790513161160093-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e4422a27f949458fbd6184e6585d7d9b"
format_stamp: "Formatted at 2026-09-27 12:50:10 on dist-test-slave-lkn4"
server_key: "045193871f5756c8a1e96372b3890c65"
server_key_iv: "58c0ddcdb8feea7936af4e6a74c49a58"
server_key_version: "encryptionkey@0"
I20260927 12:50:10.922858 19247 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 12:50:10.925541 19261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:10.927030 19247 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 12:50:10.927146 19247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790513161160093-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790513161160093-392-0/minicluster-data/ts-1/wal
uuid: "e4422a27f949458fbd6184e6585d7d9b"
format_stamp: "Formatted at 2026-09-27 12:50:10 on dist-test-slave-lkn4"
server_key: "045193871f5756c8a1e96372b3890c65"
server_key_iv: "58c0ddcdb8feea7936af4e6a74c49a58"
server_key_version: "encryptionkey@0"
I20260927 12:50:10.927275 19247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790513161160093-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790513161160093-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790513161160093-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:50:10.946487 19247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:50:10.947171 19247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:50:10.947336 19247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:50:10.947901 19247 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:50:10.948843 19247 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:50:10.948890 19247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:10.948930 19247 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:50:10.948944 19247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:10.957731 19247 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36469
I20260927 12:50:10.957913 19374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36469 every 8 connection(s)
I20260927 12:50:10.958751 19247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:50:10.966461 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 19247
I20260927 12:50:10.966570 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790513161160093-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:50:10.966897 392 external_mini_cluster.cc:1428] Setting key 2e7bb9ad357d7ce28bc3495899a3264f
I20260927 12:50:10.968457 19095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:10.960658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34659 (local address 127.0.98.62:42299)
0927 12:50:10.960818 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:10.960822 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:10.962101 (+ 1279us) server_negotiation.cc:407] Connection header received
0927 12:50:10.963506 (+ 1405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:10.963509 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:10.963567 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:10.963643 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:10.964594 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:10.965134 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:10.966386 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:10.966654 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:10.966753 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:10.967988 (+ 1235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:10.967992 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:10.967995 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:50:10.968008 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:10.968032 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:50:10.968042 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:10.968124 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:10.968262 (+ 138us) server_negotiation.cc:299] Negotiation successful
0927 12:50:10.968320 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 12:50:10.969535 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790513161160093-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790513161160093-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790513161160093-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42299
--builtin_ntp_servers=127.0.98.20:33297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 12:50:10.969736 19377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:10.961143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42299 (local address 127.0.98.2:34659)
0927 12:50:10.961916 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:10.961972 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 12:50:10.963294 (+ 1322us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:50:10.963855 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:50:10.963862 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:50:10.963952 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:50:10.964458 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:10.964470 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:10.965257 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:10.965260 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:10.966267 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:10.966276 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:10.966809 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:10.966813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:10.966981 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:10.967633 (+ 652us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 12:50:10.967635 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:50:10.967696 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 12:50:10.967788 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:50:10.968135 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:50:10.968141 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:50:10.968145 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 12:50:10.968247 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 12:50:10.968532 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":113,"threads_started":1}
I20260927 12:50:10.970942 19375 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42299
I20260927 12:50:10.971177 19375 heartbeater.cc:464] Registering TS with master...
I20260927 12:50:10.971767 19375 heartbeater.cc:511] Master 127.0.98.62:42299 requested a full tablet report, sending...
I20260927 12:50:10.972818 19056 ts_manager.cc:194] Registered new tserver with Master: e4422a27f949458fbd6184e6585d7d9b (127.0.98.2:36469)
I20260927 12:50:10.973604 19056 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:34659
W20260927 12:50:11.079618 19378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:50:11.079900 19378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:50:11.079937 19378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:50:11.083957 19378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:50:11.084023 19378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:50:11.084077 19378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:50:11.088831 19378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790513161160093-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790513161160093-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42299
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19378
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790513161160093-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:50:11.089960 19378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:50:11.091367 19378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:50:11.101043 19383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:50:11.101104 19378 server_base.cc:1034] running on GCE node
W20260927 12:50:11.101089 19386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:50:11.101043 19384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:50:11.101701 19378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:50:11.102376 19378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:50:11.103547 19378 hybrid_clock.cc:648] HybridClock initialized: now 1790513411103486 us; error 73 us; skew 500 ppm
I20260927 12:50:11.105543 19378 webserver.cc:460] Webserver started at http://127.0.98.3:46833/ using document root <none> and password file <none>
I20260927 12:50:11.106180 19378 fs_manager.cc:359] Metadata directory not provided
I20260927 12:50:11.106253 19378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:50:11.106454 19378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:50:11.108574 19378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790513161160093-392-0/minicluster-data/ts-2/data/instance:
uuid: "6a335ef57d42448880aa00f220da8588"
format_stamp: "Formatted at 2026-09-27 12:50:11 on dist-test-slave-lkn4"
server_key: "f24279633a57b01fe3d5577b6d9833d4"
server_key_iv: "ef6076532527b43c48cbe3f28dddd9d0"
server_key_version: "encryptionkey@0"
I20260927 12:50:11.109133 19378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790513161160093-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6a335ef57d42448880aa00f220da8588"
format_stamp: "Formatted at 2026-09-27 12:50:11 on dist-test-slave-lkn4"
server_key: "f24279633a57b01fe3d5577b6d9833d4"
server_key_iv: "ef6076532527b43c48cbe3f28dddd9d0"
server_key_version: "encryptionkey@0"
I20260927 12:50:11.113119 19378 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 12:50:11.115760 19392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:11.117051 19378 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 12:50:11.117165 19378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790513161160093-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790513161160093-392-0/minicluster-data/ts-2/wal
uuid: "6a335ef57d42448880aa00f220da8588"
format_stamp: "Formatted at 2026-09-27 12:50:11 on dist-test-slave-lkn4"
server_key: "f24279633a57b01fe3d5577b6d9833d4"
server_key_iv: "ef6076532527b43c48cbe3f28dddd9d0"
server_key_version: "encryptionkey@0"
I20260927 12:50:11.117283 19378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790513161160093-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790513161160093-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790513161160093-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:50:11.140295 19378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:50:11.141080 19378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:50:11.141299 19378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:50:11.141948 19378 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:50:11.142966 19378 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:50:11.143019 19378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:11.143077 19378 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:50:11.143110 19378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:11.152288 19378 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33625
I20260927 12:50:11.152319 19505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33625 every 8 connection(s)
I20260927 12:50:11.153254 19378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:50:11.156064 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 19378
I20260927 12:50:11.156178 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790513161160093-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:50:11.156442 392 external_mini_cluster.cc:1428] Setting key d8685349107d9a35c9ff7d5147b219fe
I20260927 12:50:11.162618 19095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:11.155244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36151 (local address 127.0.98.62:42299)
0927 12:50:11.155368 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:11.155372 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:11.156584 (+ 1212us) server_negotiation.cc:407] Connection header received
0927 12:50:11.158072 (+ 1488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:11.158075 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:11.158126 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:11.158218 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:11.159153 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:11.159680 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:11.160833 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:11.161079 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:11.161171 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:11.162177 (+ 1006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:11.162180 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:11.162182 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:50:11.162193 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:11.162218 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:50:11.162226 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:11.162283 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:11.162447 (+ 164us) server_negotiation.cc:299] Negotiation successful
0927 12:50:11.162502 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 12:50:11.163390 19508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:11.155592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42299 (local address 127.0.98.3:36151)
0927 12:50:11.156413 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:11.156465 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 12:50:11.157863 (+ 1398us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:50:11.158383 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:50:11.158390 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:50:11.158477 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:50:11.159032 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:11.159046 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:11.159807 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:11.159810 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:11.160678 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:11.160686 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:11.161206 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:11.161208 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:11.161361 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:11.161917 (+ 556us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 12:50:11.161918 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:50:11.161958 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 12:50:11.162021 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:50:11.162292 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:50:11.162298 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:50:11.162302 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 12:50:11.162395 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 12:50:11.162621 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":184,"threads_started":1}
I20260927 12:50:11.164474 19506 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42299
I20260927 12:50:11.164695 19506 heartbeater.cc:464] Registering TS with master...
I20260927 12:50:11.165201 19506 heartbeater.cc:511] Master 127.0.98.62:42299 requested a full tablet report, sending...
I20260927 12:50:11.166311 19056 ts_manager.cc:194] Registered new tserver with Master: 6a335ef57d42448880aa00f220da8588 (127.0.98.3:33625)
I20260927 12:50:11.166903 19056 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:36151
I20260927 12:50:11.170419 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 12:50:11.176419 19095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:11.172297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37392 (local address 127.0.98.62:42299)
0927 12:50:11.172448 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:11.172452 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:11.172549 (+ 97us) server_negotiation.cc:407] Connection header received
0927 12:50:11.172732 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:11.172734 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:11.172777 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:11.172847 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:11.173127 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:11.173641 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:11.174635 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:11.174869 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:11.174955 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:11.175869 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:11.175872 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:11.175875 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:50:11.175886 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:11.175905 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:50:11.175913 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:11.176017 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:11.176220 (+ 203us) server_negotiation.cc:299] Negotiation successful
0927 12:50:11.176273 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 12:50:11.179443 19056 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37392:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:50:11.182577 19056 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:50:11.197108 19518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:11.191136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33377 (local address 127.0.0.1:39406)
0927 12:50:11.191628 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:11.191676 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 12:50:11.191815 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:50:11.193130 (+ 1315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:50:11.193132 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:50:11.193159 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:50:11.193236 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:11.193242 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:11.194657 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:11.194659 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:11.195665 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:11.195674 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:11.196668 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:11.196672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:11.196756 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:11.196778 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 12:50:11.196835 (+ 57us) client_negotiation.cc:240] Negotiation successful
0927 12:50:11.196903 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":5,"thread_start_us":73,"threads_started":1}
I20260927 12:50:11.197139 19517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:11.191274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33625 (local address 127.0.0.1:42084)
0927 12:50:11.191749 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:11.191765 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:50:11.191850 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:50:11.192831 (+ 981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:50:11.192836 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:50:11.192860 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:50:11.193009 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:11.193017 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:11.194485 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:11.194488 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:11.195649 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:11.195656 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:11.196741 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:11.196744 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:11.196836 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:11.196858 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 12:50:11.196921 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 12:50:11.196980 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":58,"thread_start_us":107,"threads_started":1}
I20260927 12:50:11.197135 19519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:11.191304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39406 (local address 127.0.98.1:33377)
0927 12:50:11.192677 (+ 1373us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:11.192689 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:11.192712 (+ 23us) server_negotiation.cc:407] Connection header received
0927 12:50:11.192786 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:11.192791 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:11.192985 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:11.193372 (+ 387us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:50:11.193504 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:11.194552 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:11.195791 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:11.196519 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:11.196664 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:11.196837 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:11.196885 (+ 48us) server_negotiation.cc:299] Negotiation successful
0927 12:50:11.196980 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1260,"thread_start_us":93,"threads_started":1}
I20260927 12:50:11.196933 19520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:11.191827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36469 (local address 127.0.0.1:59178)
0927 12:50:11.192328 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:11.192341 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:50:11.192431 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:50:11.193059 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:50:11.193062 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:50:11.193078 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:50:11.193160 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:11.193166 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:11.194089 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:11.194092 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:11.195337 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:11.195348 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:11.196380 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:11.196384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:11.196493 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:11.196537 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 12:50:11.196643 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 12:50:11.196774 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":91,"threads_started":1}
I20260927 12:50:11.197523 19521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:11.191903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42084 (local address 127.0.98.3:33625)
0927 12:50:11.192230 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:11.192243 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:11.192267 (+ 24us) server_negotiation.cc:407] Connection header received
0927 12:50:11.192344 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:11.192351 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:11.192584 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:11.192750 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:50:11.193142 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:11.194365 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:11.195781 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:11.196404 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:11.196499 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:11.196674 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:11.197257 (+ 583us) server_negotiation.cc:299] Negotiation successful
0927 12:50:11.197366 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":99,"threads_started":1}
I20260927 12:50:11.197649 19522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:11.192238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59178 (local address 127.0.98.2:36469)
0927 12:50:11.192654 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:11.192666 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:11.192687 (+ 21us) server_negotiation.cc:407] Connection header received
0927 12:50:11.192760 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:11.192764 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:11.192934 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:11.193060 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:50:11.193269 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:11.193979 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:11.195457 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:11.196253 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:11.196359 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:11.196554 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:11.196994 (+ 440us) server_negotiation.cc:299] Negotiation successful
0927 12:50:11.197081 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":125,"threads_started":1}
I20260927 12:50:11.200294 19178 tablet_service.cc:1467] Processing CreateTablet for tablet d734a79dc827471f9e6d5ad004e3beee (DEFAULT_TABLE table=test-table [id=bd116a808b1b4a3785c4dd1c78992831]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:50:11.200346 19309 tablet_service.cc:1467] Processing CreateTablet for tablet d734a79dc827471f9e6d5ad004e3beee (DEFAULT_TABLE table=test-table [id=bd116a808b1b4a3785c4dd1c78992831]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:50:11.200349 19440 tablet_service.cc:1467] Processing CreateTablet for tablet d734a79dc827471f9e6d5ad004e3beee (DEFAULT_TABLE table=test-table [id=bd116a808b1b4a3785c4dd1c78992831]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:50:11.201269 19178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d734a79dc827471f9e6d5ad004e3beee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:50:11.201315 19440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d734a79dc827471f9e6d5ad004e3beee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:50:11.201315 19309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d734a79dc827471f9e6d5ad004e3beee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:50:11.208022 19523 tablet_bootstrap.cc:492] T d734a79dc827471f9e6d5ad004e3beee P 29daecf51a094db79140803abd92f8aa: Bootstrap starting.
I20260927 12:50:11.208526 19525 tablet_bootstrap.cc:492] T d734a79dc827471f9e6d5ad004e3beee P 6a335ef57d42448880aa00f220da8588: Bootstrap starting.
I20260927 12:50:11.210321 19524 tablet_bootstrap.cc:492] T d734a79dc827471f9e6d5ad004e3beee P e4422a27f949458fbd6184e6585d7d9b: Bootstrap starting.
I20260927 12:50:11.210719 19525 tablet_bootstrap.cc:654] T d734a79dc827471f9e6d5ad004e3beee P 6a335ef57d42448880aa00f220da8588: Neither blocks nor log segments found. Creating new log.
I20260927 12:50:11.210719 19523 tablet_bootstrap.cc:654] T d734a79dc827471f9e6d5ad004e3beee P 29daecf51a094db79140803abd92f8aa: Neither blocks nor log segments found. Creating new log.
I20260927 12:50:11.211586 19523 log.cc:824] T d734a79dc827471f9e6d5ad004e3beee P 29daecf51a094db79140803abd92f8aa: Log is configured to *not* fsync() on all Append() calls
I20260927 12:50:11.211588 19525 log.cc:824] T d734a79dc827471f9e6d5ad004e3beee P 6a335ef57d42448880aa00f220da8588: Log is configured to *not* fsync() on all Append() calls
I20260927 12:50:11.212117 19524 tablet_bootstrap.cc:654] T d734a79dc827471f9e6d5ad004e3beee P e4422a27f949458fbd6184e6585d7d9b: Neither blocks nor log segments found. Creating new log.
I20260927 12:50:11.212954 19524 log.cc:824] T d734a79dc827471f9e6d5ad004e3beee P e4422a27f949458fbd6184e6585d7d9b: Log is configured to *not* fsync() on all Append() calls
I20260927 12:50:11.213667 19525 tablet_bootstrap.cc:492] T d734a79dc827471f9e6d5ad004e3beee P 6a335ef57d42448880aa00f220da8588: No bootstrap required, opened a new log
I20260927 12:50:11.213728 19523 tablet_bootstrap.cc:492] T d734a79dc827471f9e6d5ad004e3beee P 29daecf51a094db79140803abd92f8aa: No bootstrap required, opened a new log
I20260927 12:50:11.213853 19525 ts_tablet_manager.cc:1397] T d734a79dc827471f9e6d5ad004e3beee P 6a335ef57d42448880aa00f220da8588: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 12:50:11.213924 19523 ts_tablet_manager.cc:1397] T d734a79dc827471f9e6d5ad004e3beee P 29daecf51a094db79140803abd92f8aa: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260927 12:50:11.215286 19524 tablet_bootstrap.cc:492] T d734a79dc827471f9e6d5ad004e3beee P e4422a27f949458fbd6184e6585d7d9b: No bootstrap required, opened a new log
I20260927 12:50:11.215462 19524 ts_tablet_manager.cc:1397] T d734a79dc827471f9e6d5ad004e3beee P e4422a27f949458fbd6184e6585d7d9b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 12:50:11.217307 19523 raft_consensus.cc:348] T d734a79dc827471f9e6d5ad004e3beee P 29daecf51a094db79140803abd92f8aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29daecf51a094db79140803abd92f8aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33377 } } peers { permanent_uuid: "6a335ef57d42448880aa00f220da8588" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33625 } } peers { permanent_uuid: "e4422a27f949458fbd6184e6585d7d9b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36469 } }
I20260927 12:50:11.217540 19523 raft_consensus.cc:374] T d734a79dc827471f9e6d5ad004e3beee P 29daecf51a094db79140803abd92f8aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:50:11.217602 19523 raft_consensus.cc:729] T d734a79dc827471f9e6d5ad004e3beee P 29daecf51a094db79140803abd92f8aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29daecf51a094db79140803abd92f8aa, State: Initialized, Role: FOLLOWER
I20260927 12:50:11.217923 19525 raft_consensus.cc:348] T d734a79dc827471f9e6d5ad004e3beee P 6a335ef57d42448880aa00f220da8588 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29daecf51a094db79140803abd92f8aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33377 } } peers { permanent_uuid: "6a335ef57d42448880aa00f220da8588" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33625 } } peers { permanent_uuid: "e4422a27f949458fbd6184e6585d7d9b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36469 } }
I20260927 12:50:11.218065 19523 consensus_queue.cc:260] T d734a79dc827471f9e6d5ad004e3beee P 29daecf51a094db79140803abd92f8aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29daecf51a094db79140803abd92f8aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33377 } } peers { permanent_uuid: "6a335ef57d42448880aa00f220da8588" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33625 } } peers { permanent_uuid: "e4422a27f949458fbd6184e6585d7d9b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36469 } }
I20260927 12:50:11.218199 19525 raft_consensus.cc:374] T d734a79dc827471f9e6d5ad004e3beee P 6a335ef57d42448880aa00f220da8588 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:50:11.218262 19525 raft_consensus.cc:729] T d734a79dc827471f9e6d5ad004e3beee P 6a335ef57d42448880aa00f220da8588 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a335ef57d42448880aa00f220da8588, State: Initialized, Role: FOLLOWER
I20260927 12:50:11.218660 19525 consensus_queue.cc:260] T d734a79dc827471f9e6d5ad004e3beee P 6a335ef57d42448880aa00f220da8588 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29daecf51a094db79140803abd92f8aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33377 } } peers { permanent_uuid: "6a335ef57d42448880aa00f220da8588" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33625 } } peers { permanent_uuid: "e4422a27f949458fbd6184e6585d7d9b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36469 } }
I20260927 12:50:11.218740 19244 heartbeater.cc:503] Master 127.0.98.62:42299 was elected leader, sending a full tablet report...
I20260927 12:50:11.219167 19523 ts_tablet_manager.cc:1428] T d734a79dc827471f9e6d5ad004e3beee P 29daecf51a094db79140803abd92f8aa: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 12:50:11.219583 19525 ts_tablet_manager.cc:1428] T d734a79dc827471f9e6d5ad004e3beee P 6a335ef57d42448880aa00f220da8588: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 12:50:11.219460 19524 raft_consensus.cc:348] T d734a79dc827471f9e6d5ad004e3beee P e4422a27f949458fbd6184e6585d7d9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29daecf51a094db79140803abd92f8aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33377 } } peers { permanent_uuid: "6a335ef57d42448880aa00f220da8588" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33625 } } peers { permanent_uuid: "e4422a27f949458fbd6184e6585d7d9b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36469 } }
I20260927 12:50:11.219679 19524 raft_consensus.cc:374] T d734a79dc827471f9e6d5ad004e3beee P e4422a27f949458fbd6184e6585d7d9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:50:11.219734 19524 raft_consensus.cc:729] T d734a79dc827471f9e6d5ad004e3beee P e4422a27f949458fbd6184e6585d7d9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4422a27f949458fbd6184e6585d7d9b, State: Initialized, Role: FOLLOWER
I20260927 12:50:11.219720 19506 heartbeater.cc:503] Master 127.0.98.62:42299 was elected leader, sending a full tablet report...
I20260927 12:50:11.220122 19524 consensus_queue.cc:260] T d734a79dc827471f9e6d5ad004e3beee P e4422a27f949458fbd6184e6585d7d9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29daecf51a094db79140803abd92f8aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33377 } } peers { permanent_uuid: "6a335ef57d42448880aa00f220da8588" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33625 } } peers { permanent_uuid: "e4422a27f949458fbd6184e6585d7d9b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36469 } }
I20260927 12:50:11.221042 19524 ts_tablet_manager.cc:1428] T d734a79dc827471f9e6d5ad004e3beee P e4422a27f949458fbd6184e6585d7d9b: Time spent starting tablet: real 0.006s user 0.002s sys 0.005s
I20260927 12:50:11.221341 19375 heartbeater.cc:503] Master 127.0.98.62:42299 was elected leader, sending a full tablet report...
W20260927 12:50:11.268247 19245 tablet.cc:2369] T d734a79dc827471f9e6d5ad004e3beee P 29daecf51a094db79140803abd92f8aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:50:11.306515 19529 raft_consensus.cc:482] T d734a79dc827471f9e6d5ad004e3beee P 29daecf51a094db79140803abd92f8aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:50:11.306777 19529 raft_consensus.cc:504] T d734a79dc827471f9e6d5ad004e3beee P 29daecf51a094db79140803abd92f8aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29daecf51a094db79140803abd92f8aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33377 } } peers { permanent_uuid: "6a335ef57d42448880aa00f220da8588" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33625 } } peers { permanent_uuid: "e4422a27f949458fbd6184e6585d7d9b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36469 } }
I20260927 12:50:11.307924 19529 leader_election.cc:288] T d734a79dc827471f9e6d5ad004e3beee P 29daecf51a094db79140803abd92f8aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a335ef57d42448880aa00f220da8588 (127.0.98.3:33625), e4422a27f949458fbd6184e6585d7d9b (127.0.98.2:36469)
I20260927 12:50:11.311735 19522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:11.308313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34313 (local address 127.0.98.2:36469)
0927 12:50:11.308444 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:11.308448 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:11.308758 (+ 310us) server_negotiation.cc:407] Connection header received
0927 12:50:11.308880 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:11.308883 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:11.308925 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:11.308988 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:50:11.309238 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:11.309710 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:11.310814 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:11.311190 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:11.311280 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:11.311338 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:11.311542 (+ 204us) server_negotiation.cc:299] Negotiation successful
0927 12:50:11.311608 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 12:50:11.311743 19521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:11.308321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58751 (local address 127.0.98.3:33625)
0927 12:50:11.308482 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:11.308487 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:11.308620 (+ 133us) server_negotiation.cc:407] Connection header received
0927 12:50:11.308759 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:11.308762 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:11.308804 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:11.308862 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:50:11.309139 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:11.309660 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:11.310823 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:11.311211 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:11.311300 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:11.311356 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:11.311550 (+ 194us) server_negotiation.cc:299] Negotiation successful
0927 12:50:11.311620 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 12:50:11.311764 19532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:11.308230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36469 (local address 127.0.98.1:34313)
0927 12:50:11.308603 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:11.308614 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 12:50:11.308783 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:50:11.309006 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:50:11.309008 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:50:11.309023 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:50:11.309129 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:11.309134 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:11.309807 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:11.309810 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:11.310707 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:11.310716 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:11.311298 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:11.311301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:11.311433 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:11.311458 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 12:50:11.311519 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 12:50:11.311584 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"mutex_wait_us":78,"thread_start_us":100,"threads_started":1}
I20260927 12:50:11.312119 19533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:11.308167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33625 (local address 127.0.98.1:58751)
0927 12:50:11.308537 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:11.308549 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:50:11.308657 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:50:11.308880 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:50:11.308882 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:50:11.308897 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:50:11.309033 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:11.309039 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:11.309755 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:11.309759 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:11.310702 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:11.310711 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:11.311312 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:11.311316 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:11.311445 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:11.311461 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 12:50:11.311737 (+ 276us) client_negotiation.cc:240] Negotiation successful
0927 12:50:11.311988 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":65,"thread_start_us":111,"threads_started":1}
I20260927 12:50:11.312568 19329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d734a79dc827471f9e6d5ad004e3beee" candidate_uuid: "29daecf51a094db79140803abd92f8aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4422a27f949458fbd6184e6585d7d9b" is_pre_election: true
I20260927 12:50:11.312762 19460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d734a79dc827471f9e6d5ad004e3beee" candidate_uuid: "29daecf51a094db79140803abd92f8aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a335ef57d42448880aa00f220da8588" is_pre_election: true
I20260927 12:50:11.312889 19329 raft_consensus.cc:2454] T d734a79dc827471f9e6d5ad004e3beee P e4422a27f949458fbd6184e6585d7d9b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29daecf51a094db79140803abd92f8aa in term 0.
I20260927 12:50:11.312997 19460 raft_consensus.cc:2454] T d734a79dc827471f9e6d5ad004e3beee P 6a335ef57d42448880aa00f220da8588 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29daecf51a094db79140803abd92f8aa in term 0.
I20260927 12:50:11.313352 19132 leader_election.cc:302] T d734a79dc827471f9e6d5ad004e3beee P 29daecf51a094db79140803abd92f8aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29daecf51a094db79140803abd92f8aa, e4422a27f949458fbd6184e6585d7d9b; no voters:
I20260927 12:50:11.313582 19529 raft_consensus.cc:2781] T d734a79dc827471f9e6d5ad004e3beee P 29daecf51a094db79140803abd92f8aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:50:11.313669 19529 raft_consensus.cc:482] T d734a79dc827471f9e6d5ad004e3beee P 29daecf51a094db79140803abd92f8aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:50:11.313710 19529 raft_consensus.cc:3037] T d734a79dc827471f9e6d5ad004e3beee P 29daecf51a094db79140803abd92f8aa [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:50:11.314713 19529 raft_consensus.cc:504] T d734a79dc827471f9e6d5ad004e3beee P 29daecf51a094db79140803abd92f8aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29daecf51a094db79140803abd92f8aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33377 } } peers { permanent_uuid: "6a335ef57d42448880aa00f220da8588" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33625 } } peers { permanent_uuid: "e4422a27f949458fbd6184e6585d7d9b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36469 } }
I20260927 12:50:11.315032 19529 leader_election.cc:288] T d734a79dc827471f9e6d5ad004e3beee P 29daecf51a094db79140803abd92f8aa [CANDIDATE]: Term 1 election: Requested vote from peers 6a335ef57d42448880aa00f220da8588 (127.0.98.3:33625), e4422a27f949458fbd6184e6585d7d9b (127.0.98.2:36469)
I20260927 12:50:11.315357 19329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d734a79dc827471f9e6d5ad004e3beee" candidate_uuid: "29daecf51a094db79140803abd92f8aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4422a27f949458fbd6184e6585d7d9b"
I20260927 12:50:11.315356 19460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d734a79dc827471f9e6d5ad004e3beee" candidate_uuid: "29daecf51a094db79140803abd92f8aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a335ef57d42448880aa00f220da8588"
I20260927 12:50:11.315486 19460 raft_consensus.cc:3037] T d734a79dc827471f9e6d5ad004e3beee P 6a335ef57d42448880aa00f220da8588 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:50:11.315480 19329 raft_consensus.cc:3037] T d734a79dc827471f9e6d5ad004e3beee P e4422a27f949458fbd6184e6585d7d9b [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:50:11.316735 19329 raft_consensus.cc:2454] T d734a79dc827471f9e6d5ad004e3beee P e4422a27f949458fbd6184e6585d7d9b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29daecf51a094db79140803abd92f8aa in term 1.
I20260927 12:50:11.316749 19460 raft_consensus.cc:2454] T d734a79dc827471f9e6d5ad004e3beee P 6a335ef57d42448880aa00f220da8588 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29daecf51a094db79140803abd92f8aa in term 1.
I20260927 12:50:11.317039 19131 leader_election.cc:302] T d734a79dc827471f9e6d5ad004e3beee P 29daecf51a094db79140803abd92f8aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29daecf51a094db79140803abd92f8aa, 6a335ef57d42448880aa00f220da8588; no voters:
I20260927 12:50:11.317199 19529 raft_consensus.cc:2781] T d734a79dc827471f9e6d5ad004e3beee P 29daecf51a094db79140803abd92f8aa [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:50:11.317426 19529 raft_consensus.cc:686] T d734a79dc827471f9e6d5ad004e3beee P 29daecf51a094db79140803abd92f8aa [term 1 LEADER]: Becoming Leader. State: Replica: 29daecf51a094db79140803abd92f8aa, State: Running, Role: LEADER
I20260927 12:50:11.317759 19529 consensus_queue.cc:237] T d734a79dc827471f9e6d5ad004e3beee P 29daecf51a094db79140803abd92f8aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29daecf51a094db79140803abd92f8aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33377 } } peers { permanent_uuid: "6a335ef57d42448880aa00f220da8588" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33625 } } peers { permanent_uuid: "e4422a27f949458fbd6184e6585d7d9b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36469 } }
I20260927 12:50:11.320972 19055 catalog_manager.cc:5495] T d734a79dc827471f9e6d5ad004e3beee P 29daecf51a094db79140803abd92f8aa reported cstate change: term changed from 0 to 1, leader changed from <none> to 29daecf51a094db79140803abd92f8aa (127.0.98.1). New cstate: current_term: 1 leader_uuid: "29daecf51a094db79140803abd92f8aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29daecf51a094db79140803abd92f8aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a335ef57d42448880aa00f220da8588" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4422a27f949458fbd6184e6585d7d9b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36469 } health_report { overall_health: UNKNOWN } } }
I20260927 12:50:11.351365 19519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:11.347549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39408 (local address 127.0.98.1:33377)
0927 12:50:11.347744 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:11.347748 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:11.347762 (+ 14us) server_negotiation.cc:407] Connection header received
0927 12:50:11.347834 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:11.347837 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:11.347914 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:11.348016 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:50:11.348325 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:11.348893 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:11.349770 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:11.349998 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:11.350124 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:11.350404 (+ 280us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:50:11.350899 (+ 495us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:50:11.351020 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:11.351208 (+ 188us) server_negotiation.cc:299] Negotiation successful
0927 12:50:11.351250 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 12:50:11.357779 19329 raft_consensus.cc:1261] T d734a79dc827471f9e6d5ad004e3beee P e4422a27f949458fbd6184e6585d7d9b [term 1 FOLLOWER]: Refusing update from remote peer 29daecf51a094db79140803abd92f8aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:50:11.357779 19460 raft_consensus.cc:1261] T d734a79dc827471f9e6d5ad004e3beee P 6a335ef57d42448880aa00f220da8588 [term 1 FOLLOWER]: Refusing update from remote peer 29daecf51a094db79140803abd92f8aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:50:11.358522 19534 consensus_queue.cc:1035] T d734a79dc827471f9e6d5ad004e3beee P 29daecf51a094db79140803abd92f8aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4422a27f949458fbd6184e6585d7d9b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:50:11.358713 19529 consensus_queue.cc:1035] T d734a79dc827471f9e6d5ad004e3beee P 29daecf51a094db79140803abd92f8aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a335ef57d42448880aa00f220da8588" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:50:11.367096 19538 mvcc.cc:205] Tried to move back new op lower bound from 7333942932913180672 to 7333942932760100864. Current Snapshot: MvccSnapshot[applied={T|T < 7333942932913180672 or (T in {7333942932913180672})}]
I20260927 12:50:11.373474 19539 mvcc.cc:205] Tried to move back new op lower bound from 7333942932913180672 to 7333942932760100864. Current Snapshot: MvccSnapshot[applied={T|T < 7333942932913180672 or (T in {7333942932913180672})}]
I20260927 12:50:11.374511 19055 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:50:11.374694 19055 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37392:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:50:11.377054 19055 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 75b2ac1e75cf46d68cb6ab0af3797758: Sending DeleteTablet for 3 replicas of tablet d734a79dc827471f9e6d5ad004e3beee
I20260927 12:50:11.377828 19440 tablet_service.cc:1514] Processing DeleteTablet for tablet d734a79dc827471f9e6d5ad004e3beee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:50:11 UTC) from {username='slave'} at 127.0.0.1:42084
I20260927 12:50:11.377828 19309 tablet_service.cc:1514] Processing DeleteTablet for tablet d734a79dc827471f9e6d5ad004e3beee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:50:11 UTC) from {username='slave'} at 127.0.0.1:59178
I20260927 12:50:11.378271 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 19116
I20260927 12:50:11.378357 19178 tablet_service.cc:1514] Processing DeleteTablet for tablet d734a79dc827471f9e6d5ad004e3beee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:50:11 UTC) from {username='slave'} at 127.0.0.1:39406
I20260927 12:50:11.379045 19548 tablet_replica.cc:330] T d734a79dc827471f9e6d5ad004e3beee P e4422a27f949458fbd6184e6585d7d9b: stopping tablet replica
I20260927 12:50:11.379371 19548 raft_consensus.cc:2229] T d734a79dc827471f9e6d5ad004e3beee P e4422a27f949458fbd6184e6585d7d9b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:50:11.379783 19547 tablet_replica.cc:330] T d734a79dc827471f9e6d5ad004e3beee P 6a335ef57d42448880aa00f220da8588: stopping tablet replica
I20260927 12:50:11.380079 19547 raft_consensus.cc:2229] T d734a79dc827471f9e6d5ad004e3beee P 6a335ef57d42448880aa00f220da8588 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:50:11.380378 19547 raft_consensus.cc:2258] T d734a79dc827471f9e6d5ad004e3beee P 6a335ef57d42448880aa00f220da8588 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:50:11.380800 19548 raft_consensus.cc:2258] T d734a79dc827471f9e6d5ad004e3beee P e4422a27f949458fbd6184e6585d7d9b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:50:11.382241 19548 ts_tablet_manager.cc:1905] T d734a79dc827471f9e6d5ad004e3beee P e4422a27f949458fbd6184e6585d7d9b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:50:11.383319 19547 ts_tablet_manager.cc:1905] T d734a79dc827471f9e6d5ad004e3beee P 6a335ef57d42448880aa00f220da8588: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:50:11.385025 19548 ts_tablet_manager.cc:1918] T d734a79dc827471f9e6d5ad004e3beee P e4422a27f949458fbd6184e6585d7d9b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:50:11.385133 19548 log.cc:1196] T d734a79dc827471f9e6d5ad004e3beee P e4422a27f949458fbd6184e6585d7d9b: Deleting WAL directory at /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790513161160093-392-0/minicluster-data/ts-1/wal/wals/d734a79dc827471f9e6d5ad004e3beee
W20260927 12:50:11.385241 19041 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33377 (error 108)
I20260927 12:50:11.385432 19548 ts_tablet_manager.cc:1939] T d734a79dc827471f9e6d5ad004e3beee P e4422a27f949458fbd6184e6585d7d9b: Deleting consensus metadata
I20260927 12:50:11.386137 19547 ts_tablet_manager.cc:1918] T d734a79dc827471f9e6d5ad004e3beee P 6a335ef57d42448880aa00f220da8588: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:50:11.386158 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 19247
I20260927 12:50:11.386237 19547 log.cc:1196] T d734a79dc827471f9e6d5ad004e3beee P 6a335ef57d42448880aa00f220da8588: Deleting WAL directory at /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790513161160093-392-0/minicluster-data/ts-2/wal/wals/d734a79dc827471f9e6d5ad004e3beee
I20260927 12:50:11.386724 19547 ts_tablet_manager.cc:1939] T d734a79dc827471f9e6d5ad004e3beee P 6a335ef57d42448880aa00f220da8588: Deleting consensus metadata
I20260927 12:50:11.387189 19517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:11.386476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33377 (local address 127.0.0.1:39414)
0927 12:50:11.387016 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:11.387088 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449}
W20260927 12:50:11.387406 19041 catalog_manager.cc:4584] TS 29daecf51a094db79140803abd92f8aa (127.0.98.1:33377): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d734a79dc827471f9e6d5ad004e3beee: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33377: connect: Connection refused (error 111)
I20260927 12:50:11.387950 19040 catalog_manager.cc:4841] TS 6a335ef57d42448880aa00f220da8588 (127.0.98.3:33625): tablet d734a79dc827471f9e6d5ad004e3beee (table test-table [id=bd116a808b1b4a3785c4dd1c78992831]) successfully deleted
I20260927 12:50:11.391598 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 19378
I20260927 12:50:11.392696 19517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:11.391819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36469 (local address 127.0.0.1:59182)
0927 12:50:11.392572 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:11.392621 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:36469: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":676}
I20260927 12:50:11.396972 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 19025
2026-09-27T12:50:11Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1099 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:50:11 dist-test-slave-lkn4 krb5kdc[19552](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:50:11 dist-test-slave-lkn4 krb5kdc[19552](info): set up 2 sockets
Sep 27 12:50:11 dist-test-slave-lkn4 krb5kdc[19552](info): commencing operation
krb5kdc: starting...
W20260927 12:50:12.354153 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/krb5kdc/test-user.keytab.
Sep 27 12:50:12 dist-test-slave-lkn4 krb5kdc[19552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:50:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:50:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:50:12.477460 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35029
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35029
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:50:12.586758 19565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:50:12.587050 19565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:50:12.587109 19565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:50:12.591145 19565 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:50:12.591217 19565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:50:12.591244 19565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:50:12.591265 19565 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:50:12.591286 19565 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:50:12.596725 19565 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35029
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19565
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:50:12.597841 19565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:50:12.598906 19565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 12:50:12.607620 19565 server_base.cc:1034] running on GCE node
W20260927 12:50:12.607577 19570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:50:12.607577 19571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:50:12.607573 19573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:50:12.608453 19565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:50:12.609627 19565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:50:12.610792 19565 hybrid_clock.cc:648] HybridClock initialized: now 1790513412610768 us; error 46 us; skew 500 ppm
Sep 27 12:50:12 dist-test-slave-lkn4 krb5kdc[19552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513412, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:50:12.613626 19565 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:50:12.614557 19565 webserver.cc:460] Webserver started at http://127.0.98.62:44139/ using document root <none> and password file <none>
I20260927 12:50:12.615100 19565 fs_manager.cc:359] Metadata directory not provided
I20260927 12:50:12.615161 19565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:50:12.615329 19565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:50:12.617421 19565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/minicluster-data/master-0/data/instance:
uuid: "99a6bc1e2fda4f17ab8b6344622708e7"
format_stamp: "Formatted at 2026-09-27 12:50:12 on dist-test-slave-lkn4"
server_key: "499627c3797555f762361a537c1bad3b"
server_key_iv: "a6c91126db16e160a1e1a884f269ad04"
server_key_version: "encryptionkey@0"
I20260927 12:50:12.617954 19565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/minicluster-data/master-0/wal/instance:
uuid: "99a6bc1e2fda4f17ab8b6344622708e7"
format_stamp: "Formatted at 2026-09-27 12:50:12 on dist-test-slave-lkn4"
server_key: "499627c3797555f762361a537c1bad3b"
server_key_iv: "a6c91126db16e160a1e1a884f269ad04"
server_key_version: "encryptionkey@0"
I20260927 12:50:12.621671 19565 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 12:50:12.624305 19580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:12.625546 19565 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 12:50:12.625651 19565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/minicluster-data/master-0/wal
uuid: "99a6bc1e2fda4f17ab8b6344622708e7"
format_stamp: "Formatted at 2026-09-27 12:50:12 on dist-test-slave-lkn4"
server_key: "499627c3797555f762361a537c1bad3b"
server_key_iv: "a6c91126db16e160a1e1a884f269ad04"
server_key_version: "encryptionkey@0"
I20260927 12:50:12.625747 19565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:50:12.639695 19565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:50:12.643062 19565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:50:12.643232 19565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:50:12.650329 19565 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35029
I20260927 12:50:12.650365 19632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35029 every 8 connection(s)
I20260927 12:50:12.651346 19565 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:50:12.654191 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 19565
I20260927 12:50:12.654311 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/minicluster-data/master-0/wal/instance
I20260927 12:50:12.654415 19633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:50:12.654594 392 external_mini_cluster.cc:1428] Setting key 63bc0de9535f7fdd481c307956318711
I20260927 12:50:12.660956 19633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99a6bc1e2fda4f17ab8b6344622708e7: Bootstrap starting.
Sep 27 12:50:12 dist-test-slave-lkn4 krb5kdc[19552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:50:12.663735 19633 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99a6bc1e2fda4f17ab8b6344622708e7: Neither blocks nor log segments found. Creating new log.
I20260927 12:50:12.664585 19633 log.cc:824] T 00000000000000000000000000000000 P 99a6bc1e2fda4f17ab8b6344622708e7: Log is configured to *not* fsync() on all Append() calls
I20260927 12:50:12.666877 19633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99a6bc1e2fda4f17ab8b6344622708e7: No bootstrap required, opened a new log
I20260927 12:50:12.669170 19636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:12.656077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58548 (local address 127.0.98.62:35029)
0927 12:50:12.656535 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:12.656549 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:12.656587 (+ 38us) server_negotiation.cc:407] Connection header received
0927 12:50:12.657215 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:12.657233 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:12.657619 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:12.657952 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:12.658765 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:12.659754 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:12.660589 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:12.661061 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:12.661205 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:12.663906 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:12.663911 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:12.663925 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:50:12.663956 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:12.665887 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:12.666350 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:12.666356 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:12.666363 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:12.666506 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:12.666773 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:12.666776 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:12.666778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:12.667319 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:12.667507 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:12.667817 (+ 310us) server_negotiation.cc:299] Negotiation successful
0927 12:50:12.668126 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":156,"threads_started":1}
I20260927 12:50:12.671255 19633 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99a6bc1e2fda4f17ab8b6344622708e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99a6bc1e2fda4f17ab8b6344622708e7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35029 } }
I20260927 12:50:12.671525 19633 raft_consensus.cc:374] T 00000000000000000000000000000000 P 99a6bc1e2fda4f17ab8b6344622708e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:50:12.671586 19633 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99a6bc1e2fda4f17ab8b6344622708e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99a6bc1e2fda4f17ab8b6344622708e7, State: Initialized, Role: FOLLOWER
I20260927 12:50:12.671988 19633 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99a6bc1e2fda4f17ab8b6344622708e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99a6bc1e2fda4f17ab8b6344622708e7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35029 } }
I20260927 12:50:12.672106 19633 raft_consensus.cc:388] T 00000000000000000000000000000000 P 99a6bc1e2fda4f17ab8b6344622708e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:50:12.672149 19633 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99a6bc1e2fda4f17ab8b6344622708e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:50:12.672227 19633 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99a6bc1e2fda4f17ab8b6344622708e7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:50:12.673121 19633 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99a6bc1e2fda4f17ab8b6344622708e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99a6bc1e2fda4f17ab8b6344622708e7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35029 } }
I20260927 12:50:12.673429 19633 leader_election.cc:302] T 00000000000000000000000000000000 P 99a6bc1e2fda4f17ab8b6344622708e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99a6bc1e2fda4f17ab8b6344622708e7; no voters:
I20260927 12:50:12.673705 19633 leader_election.cc:288] T 00000000000000000000000000000000 P 99a6bc1e2fda4f17ab8b6344622708e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:50:12.673815 19638 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99a6bc1e2fda4f17ab8b6344622708e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:50:12.674368 19638 raft_consensus.cc:686] T 00000000000000000000000000000000 P 99a6bc1e2fda4f17ab8b6344622708e7 [term 1 LEADER]: Becoming Leader. State: Replica: 99a6bc1e2fda4f17ab8b6344622708e7, State: Running, Role: LEADER
I20260927 12:50:12.674675 19638 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99a6bc1e2fda4f17ab8b6344622708e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99a6bc1e2fda4f17ab8b6344622708e7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35029 } }
I20260927 12:50:12.674957 19633 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99a6bc1e2fda4f17ab8b6344622708e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:50:12.676278 19640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99a6bc1e2fda4f17ab8b6344622708e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99a6bc1e2fda4f17ab8b6344622708e7. Latest consensus state: current_term: 1 leader_uuid: "99a6bc1e2fda4f17ab8b6344622708e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99a6bc1e2fda4f17ab8b6344622708e7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35029 } } }
I20260927 12:50:12.676465 19640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99a6bc1e2fda4f17ab8b6344622708e7 [sys.catalog]: This master's current role is: LEADER
I20260927 12:50:12.676298 19639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99a6bc1e2fda4f17ab8b6344622708e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "99a6bc1e2fda4f17ab8b6344622708e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99a6bc1e2fda4f17ab8b6344622708e7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35029 } } }
I20260927 12:50:12.676563 19639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99a6bc1e2fda4f17ab8b6344622708e7 [sys.catalog]: This master's current role is: LEADER
I20260927 12:50:12.676882 19650 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:50:12.680136 19650 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:50:12.686537 19650 catalog_manager.cc:1347] Generated new cluster ID: 39de0f4a772d49899cb1587a28ba0afc
I20260927 12:50:12.686640 19650 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:50:12.706004 19650 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:50:12.707072 19650 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:50:12.713299 19650 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 99a6bc1e2fda4f17ab8b6344622708e7: Generated new TSK 0
I20260927 12:50:12.714120 19650 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 12:50:12.748669 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35029
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:50:12.861882 19659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:50:12.862198 19659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:50:12.862250 19659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:50:12.866356 19659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:50:12.866426 19659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:50:12.866487 19659 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:50:12.871867 19659 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19659
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:50:12.872987 19659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:50:12.873999 19659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:50:12.884020 19667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:50:12.884075 19665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:50:12.884053 19659 server_base.cc:1034] running on GCE node
W20260927 12:50:12.884020 19664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:50:12.884675 19659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:50:12.885298 19659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:50:12.886452 19659 hybrid_clock.cc:648] HybridClock initialized: now 1790513412886432 us; error 33 us; skew 500 ppm
Sep 27 12:50:12 dist-test-slave-lkn4 krb5kdc[19552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513412, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:50:12.889173 19659 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 12:50:12.890125 19659 webserver.cc:460] Webserver started at http://127.0.98.1:44591/ using document root <none> and password file <none>
I20260927 12:50:12.890661 19659 fs_manager.cc:359] Metadata directory not provided
I20260927 12:50:12.890735 19659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:50:12.890935 19659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:50:12.892941 19659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/minicluster-data/ts-0/data/instance:
uuid: "ff5147bcb6c84c5bae8939f8a213b5b4"
format_stamp: "Formatted at 2026-09-27 12:50:12 on dist-test-slave-lkn4"
server_key: "81ca56d3e89ff5a699a5cd6e13f0ed9b"
server_key_iv: "5146a57cf7156ba88752be60b9a3d019"
server_key_version: "encryptionkey@0"
I20260927 12:50:12.893468 19659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ff5147bcb6c84c5bae8939f8a213b5b4"
format_stamp: "Formatted at 2026-09-27 12:50:12 on dist-test-slave-lkn4"
server_key: "81ca56d3e89ff5a699a5cd6e13f0ed9b"
server_key_iv: "5146a57cf7156ba88752be60b9a3d019"
server_key_version: "encryptionkey@0"
I20260927 12:50:12.897294 19659 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 12:50:12.899842 19674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:12.901255 19659 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 12:50:12.901379 19659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/minicluster-data/ts-0/wal
uuid: "ff5147bcb6c84c5bae8939f8a213b5b4"
format_stamp: "Formatted at 2026-09-27 12:50:12 on dist-test-slave-lkn4"
server_key: "81ca56d3e89ff5a699a5cd6e13f0ed9b"
server_key_iv: "5146a57cf7156ba88752be60b9a3d019"
server_key_version: "encryptionkey@0"
I20260927 12:50:12.901492 19659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:50:12.942270 19659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:50:12.945672 19659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:50:12.945864 19659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:50:12.946497 19659 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:50:12.947368 19659 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:50:12.947417 19659 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:12.947482 19659 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:50:12.947507 19659 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:12.956519 19659 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36119
I20260927 12:50:12.956557 19787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36119 every 8 connection(s)
I20260927 12:50:12.957468 19659 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:50:12.965911 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 19659
I20260927 12:50:12.966066 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:50:12.966344 392 external_mini_cluster.cc:1428] Setting key abe07cf9c2b5df8cb38fe74439dac7b1
Sep 27 12:50:12 dist-test-slave-lkn4 krb5kdc[19552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513412, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:50:12.971261 19636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:12.959389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39841 (local address 127.0.98.62:35029)
0927 12:50:12.959532 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:12.959536 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:12.960528 (+ 992us) server_negotiation.cc:407] Connection header received
0927 12:50:12.961419 (+ 891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:12.961422 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:12.961469 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:12.961545 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:12.962869 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:12.963438 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:12.964624 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:12.964934 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:12.965029 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:12.967618 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:12.967621 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:12.967624 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:50:12.967646 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:12.969055 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:12.969601 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:12.969605 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:12.969607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:12.969689 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:12.970019 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:12.970022 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:12.970024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:12.970282 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:12.970373 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:12.970950 (+ 577us) server_negotiation.cc:299] Negotiation successful
0927 12:50:12.971085 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 12:50:12.972214 19790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:12.959875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35029 (local address 127.0.98.1:39841)
0927 12:50:12.960362 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:12.960410 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 12:50:12.961203 (+ 793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:50:12.961705 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:50:12.961713 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:50:12.962152 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:50:12.962701 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:12.962716 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:12.963562 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:12.963565 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:12.964479 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:12.964488 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:12.965083 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:12.965085 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:12.965232 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:12.965884 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:50:12.965901 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:50:12.967456 (+ 1555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:50:12.969186 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:50:12.969203 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:50:12.969221 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:50:12.969504 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:50:12.969777 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:50:12.969780 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:50:12.969782 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:50:12.969933 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:50:12.970419 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:50:12.970425 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:50:12.970694 (+ 269us) client_negotiation.cc:769] Sending connection context
0927 12:50:12.970897 (+ 203us) client_negotiation.cc:240] Negotiation successful
0927 12:50:12.971188 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":115,"threads_started":1}
I20260927 12:50:12.973512 19788 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35029
I20260927 12:50:12.973811 19788 heartbeater.cc:464] Registering TS with master...
I20260927 12:50:12.974483 19788 heartbeater.cc:511] Master 127.0.98.62:35029 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 12:50:12.976310 19597 ts_manager.cc:194] Registered new tserver with Master: ff5147bcb6c84c5bae8939f8a213b5b4 (127.0.98.1:36119)
I20260927 12:50:12.977658 19597 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39841
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 12:50:12.993844 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35029
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:50:13.107627 19793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:50:13.107936 19793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:50:13.107998 19793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:50:13.112368 19793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:50:13.112466 19793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:50:13.112560 19793 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:50:13.118264 19793 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19793
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:50:13.119405 19793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:50:13.120443 19793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:50:13.130419 19799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:50:13.130525 19793 server_base.cc:1034] running on GCE node
W20260927 12:50:13.130538 19798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:50:13.130812 19801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:50:13.131218 19793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:50:13.131918 19793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:50:13.133092 19793 hybrid_clock.cc:648] HybridClock initialized: now 1790513413133070 us; error 33 us; skew 500 ppm
Sep 27 12:50:13 dist-test-slave-lkn4 krb5kdc[19552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513413, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:50:13.135957 19793 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 12:50:13.136946 19793 webserver.cc:460] Webserver started at http://127.0.98.2:36343/ using document root <none> and password file <none>
I20260927 12:50:13.137650 19793 fs_manager.cc:359] Metadata directory not provided
I20260927 12:50:13.137734 19793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:50:13.137952 19793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:50:13.140210 19793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/minicluster-data/ts-1/data/instance:
uuid: "2c59ae273d4743778da9457d9c77a599"
format_stamp: "Formatted at 2026-09-27 12:50:13 on dist-test-slave-lkn4"
server_key: "c32edd1b810721ac9f103c789c1c028e"
server_key_iv: "7f391190d22ff8819a9c511d8f533bd8"
server_key_version: "encryptionkey@0"
I20260927 12:50:13.140820 19793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2c59ae273d4743778da9457d9c77a599"
format_stamp: "Formatted at 2026-09-27 12:50:13 on dist-test-slave-lkn4"
server_key: "c32edd1b810721ac9f103c789c1c028e"
server_key_iv: "7f391190d22ff8819a9c511d8f533bd8"
server_key_version: "encryptionkey@0"
I20260927 12:50:13.144870 19793 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 12:50:13.147526 19808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:13.148947 19793 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 12:50:13.149070 19793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/minicluster-data/ts-1/wal
uuid: "2c59ae273d4743778da9457d9c77a599"
format_stamp: "Formatted at 2026-09-27 12:50:13 on dist-test-slave-lkn4"
server_key: "c32edd1b810721ac9f103c789c1c028e"
server_key_iv: "7f391190d22ff8819a9c511d8f533bd8"
server_key_version: "encryptionkey@0"
I20260927 12:50:13.149168 19793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:50:13.183511 19793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:50:13.186563 19793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:50:13.186736 19793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:50:13.187281 19793 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:50:13.188092 19793 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:50:13.188136 19793 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:13.188174 19793 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:50:13.188194 19793 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:13.196528 19793 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46095
I20260927 12:50:13.196548 19921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46095 every 8 connection(s)
I20260927 12:50:13.197440 19793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:50:13.201139 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 19793
I20260927 12:50:13.201252 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:50:13.201501 392 external_mini_cluster.cc:1428] Setting key e904f731ab2d0b86b53a1652b63628a4
Sep 27 12:50:13 dist-test-slave-lkn4 krb5kdc[19552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513413, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 12:50:13.212399 19636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:13.199291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38577 (local address 127.0.98.62:35029)
0927 12:50:13.199450 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:13.199455 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:13.200565 (+ 1110us) server_negotiation.cc:407] Connection header received
0927 12:50:13.201637 (+ 1072us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:13.201641 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:13.201702 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:13.201785 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:13.203163 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:13.203684 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:13.205229 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:13.205496 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:13.205580 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:13.208544 (+ 2964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:13.208548 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:13.208550 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:50:13.208573 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:13.210187 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:13.210831 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:13.210834 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:13.210835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:13.210912 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:13.211310 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:13.211313 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:13.211314 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:13.211495 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:13.211572 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:13.212119 (+ 547us) server_negotiation.cc:299] Negotiation successful
0927 12:50:13.212243 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 12:50:13.213328 19924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:13.199661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35029 (local address 127.0.98.2:38577)
0927 12:50:13.200403 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:13.200447 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 12:50:13.201403 (+ 956us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:50:13.201973 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:50:13.201980 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:50:13.202453 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:50:13.203025 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:13.203041 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:13.203810 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:13.203813 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:13.205101 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:13.205114 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:13.205669 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:13.205674 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:13.205890 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:13.206560 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:50:13.206584 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:50:13.208348 (+ 1764us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:50:13.210301 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:50:13.210308 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:50:13.210326 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:50:13.210699 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:50:13.211000 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:50:13.211004 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:50:13.211006 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:50:13.211216 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:50:13.211611 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:50:13.211617 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:50:13.211892 (+ 275us) client_negotiation.cc:769] Sending connection context
0927 12:50:13.212063 (+ 171us) client_negotiation.cc:240] Negotiation successful
0927 12:50:13.212393 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":114,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 12:50:13.214586 19922 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35029
I20260927 12:50:13.214847 19922 heartbeater.cc:464] Registering TS with master...
I20260927 12:50:13.215377 19922 heartbeater.cc:511] Master 127.0.98.62:35029 requested a full tablet report, sending...
I20260927 12:50:13.216442 19597 ts_manager.cc:194] Registered new tserver with Master: 2c59ae273d4743778da9457d9c77a599 (127.0.98.2:46095)
I20260927 12:50:13.217157 19597 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38577
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 12:50:13.228905 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35029
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:50:13.340943 19927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:50:13.341248 19927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:50:13.341293 19927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:50:13.345489 19927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:50:13.345567 19927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:50:13.345638 19927 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:50:13.351189 19927 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19927
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:50:13.352397 19927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:50:13.353474 19927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:50:13.363391 19935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:50:13.363552 19927 server_base.cc:1034] running on GCE node
W20260927 12:50:13.363426 19933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:50:13.363435 19932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:50:13.364097 19927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:50:13.364825 19927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:50:13.366008 19927 hybrid_clock.cc:648] HybridClock initialized: now 1790513413365973 us; error 46 us; skew 500 ppm
Sep 27 12:50:13 dist-test-slave-lkn4 krb5kdc[19552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513413, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:50:13.369026 19927 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 12:50:13.369963 19927 webserver.cc:460] Webserver started at http://127.0.98.3:35713/ using document root <none> and password file <none>
I20260927 12:50:13.370540 19927 fs_manager.cc:359] Metadata directory not provided
I20260927 12:50:13.370609 19927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:50:13.370831 19927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:50:13.372888 19927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/minicluster-data/ts-2/data/instance:
uuid: "203447288e9c45bc92259702580fd2af"
format_stamp: "Formatted at 2026-09-27 12:50:13 on dist-test-slave-lkn4"
server_key: "a8910ed5fb774efc0b987557419f4c0e"
server_key_iv: "67fa14c9076f2a6d7aca1f48d4870d0c"
server_key_version: "encryptionkey@0"
I20260927 12:50:13.373428 19927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/minicluster-data/ts-2/wal/instance:
uuid: "203447288e9c45bc92259702580fd2af"
format_stamp: "Formatted at 2026-09-27 12:50:13 on dist-test-slave-lkn4"
server_key: "a8910ed5fb774efc0b987557419f4c0e"
server_key_iv: "67fa14c9076f2a6d7aca1f48d4870d0c"
server_key_version: "encryptionkey@0"
I20260927 12:50:13.377308 19927 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 12:50:13.379694 19942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:13.381050 19927 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 12:50:13.381165 19927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/minicluster-data/ts-2/wal
uuid: "203447288e9c45bc92259702580fd2af"
format_stamp: "Formatted at 2026-09-27 12:50:13 on dist-test-slave-lkn4"
server_key: "a8910ed5fb774efc0b987557419f4c0e"
server_key_iv: "67fa14c9076f2a6d7aca1f48d4870d0c"
server_key_version: "encryptionkey@0"
I20260927 12:50:13.381266 19927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:50:13.395355 19927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:50:13.398917 19927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:50:13.399106 19927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:50:13.399719 19927 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:50:13.400540 19927 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:50:13.400588 19927 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:13.400638 19927 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:50:13.400666 19927 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:13.409723 19927 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39875
I20260927 12:50:13.409749 20055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39875 every 8 connection(s)
I20260927 12:50:13.410663 19927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:50:13.415488 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 19927
I20260927 12:50:13.415602 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:50:13.415918 392 external_mini_cluster.cc:1428] Setting key 82bb24ffd15d64d621b25f7d6bb56624
Sep 27 12:50:13 dist-test-slave-lkn4 krb5kdc[19552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513413, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:50:13.424925 19636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:13.412518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38911 (local address 127.0.98.62:35029)
0927 12:50:13.412723 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:13.412727 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:13.413802 (+ 1075us) server_negotiation.cc:407] Connection header received
0927 12:50:13.414758 (+ 956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:13.414761 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:13.414822 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:13.414895 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:13.416350 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:13.417087 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:13.418413 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:13.418628 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:13.418709 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:13.421333 (+ 2624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:13.421336 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:13.421338 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:50:13.421365 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:13.422847 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:13.423361 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:13.423364 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:13.423365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:13.423450 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:13.423840 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:13.423842 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:13.423843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:13.424021 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:13.424090 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:13.424675 (+ 585us) server_negotiation.cc:299] Negotiation successful
0927 12:50:13.424793 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 12:50:13.425870 20058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:13.412921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35029 (local address 127.0.98.3:38911)
0927 12:50:13.413605 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:13.413660 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 12:50:13.414539 (+ 879us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:50:13.415079 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:50:13.415086 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:50:13.415482 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:50:13.416150 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:13.416169 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:13.417200 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:13.417203 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:13.418309 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:13.418319 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:13.418815 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:13.418818 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:13.418977 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:13.419574 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:50:13.419600 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:50:13.421153 (+ 1553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:50:13.422970 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:50:13.422976 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:50:13.422991 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:50:13.423256 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:50:13.423564 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:50:13.423567 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:50:13.423568 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:50:13.423754 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:50:13.424119 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:50:13.424124 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:50:13.424398 (+ 274us) client_negotiation.cc:769] Sending connection context
0927 12:50:13.424586 (+ 188us) client_negotiation.cc:240] Negotiation successful
0927 12:50:13.424966 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":115,"threads_started":1}
I20260927 12:50:13.427361 20056 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35029
I20260927 12:50:13.427605 20056 heartbeater.cc:464] Registering TS with master...
I20260927 12:50:13.428133 20056 heartbeater.cc:511] Master 127.0.98.62:35029 requested a full tablet report, sending...
I20260927 12:50:13.429152 19597 ts_manager.cc:194] Registered new tserver with Master: 203447288e9c45bc92259702580fd2af (127.0.98.3:39875)
I20260927 12:50:13.429859 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 12:50:13.429921 19597 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38911
I20260927 12:50:13.439937 19636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:13.431914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58550 (local address 127.0.98.62:35029)
0927 12:50:13.432022 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:13.432026 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:13.432413 (+ 387us) server_negotiation.cc:407] Connection header received
0927 12:50:13.432561 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:13.432565 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:13.432609 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:13.432682 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:13.433515 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:13.434174 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:13.435033 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:13.435269 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:13.435365 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:13.436872 (+ 1507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:13.436876 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:13.436878 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:50:13.436904 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:13.438271 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:13.438684 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:13.438687 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:13.438689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:13.438769 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:13.439016 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:13.439019 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:13.439021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:13.439283 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:13.439409 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:13.439645 (+ 236us) server_negotiation.cc:299] Negotiation successful
0927 12:50:13.439771 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 12:50:13.443082 19597 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58550:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:50:13.446179 19597 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:50:13.460242 20067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:13.454888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39875 (local address 127.0.0.1:57346)
0927 12:50:13.455241 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:13.455297 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 12:50:13.455418 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:50:13.456476 (+ 1058us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:50:13.456481 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:50:13.456501 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:50:13.456615 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:13.456619 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:13.457759 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:13.457763 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:13.458942 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:13.458952 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:13.459710 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:13.459721 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:13.459823 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:13.459859 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 12:50:13.459962 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 12:50:13.460083 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":57,"thread_start_us":82,"threads_started":1}
I20260927 12:50:13.460249 20068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:13.455061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57346 (local address 127.0.98.3:39875)
0927 12:50:13.455994 (+ 933us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:13.456006 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:13.456030 (+ 24us) server_negotiation.cc:407] Connection header received
0927 12:50:13.456103 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:13.456109 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:13.456325 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:13.456491 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:50:13.456719 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:13.457648 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:13.459065 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:13.459600 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:13.459696 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:13.459846 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:13.459998 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 12:50:13.460093 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":834,"thread_start_us":89,"threads_started":1}
I20260927 12:50:13.461421 20069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:13.455149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46095 (local address 127.0.0.1:46484)
0927 12:50:13.455896 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:13.455909 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:50:13.455995 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:50:13.456848 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:50:13.456851 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:50:13.456868 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:50:13.456970 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:13.456977 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:13.458408 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:13.458412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:13.459959 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:13.459966 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:13.461030 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:13.461034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:13.461126 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:13.461149 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 12:50:13.461229 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 12:50:13.461294 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"mutex_wait_us":144,"thread_start_us":96,"threads_started":1}
I20260927 12:50:13.461444 20070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:13.455542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46484 (local address 127.0.98.2:46095)
0927 12:50:13.456429 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:13.456440 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:13.456460 (+ 20us) server_negotiation.cc:407] Connection header received
0927 12:50:13.456530 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:13.456535 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:13.456710 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:13.456850 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:50:13.457089 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:13.458264 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:13.460093 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:13.460915 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:13.461019 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:13.461149 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:13.461251 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 12:50:13.461321 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":794,"thread_start_us":129,"threads_started":1}
I20260927 12:50:13.462064 20071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:13.455613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36119 (local address 127.0.0.1:52066)
0927 12:50:13.456274 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:13.456285 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 12:50:13.456376 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:50:13.457044 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:50:13.457048 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:50:13.457065 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:50:13.457159 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:13.457164 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:13.458485 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:13.458489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:13.459956 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:13.459966 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:13.461689 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:13.461693 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:13.461776 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:13.461795 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 12:50:13.461860 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 12:50:13.461925 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"mutex_wait_us":176,"thread_start_us":74,"threads_started":1}
I20260927 12:50:13.462342 20072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:13.455791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52066 (local address 127.0.98.1:36119)
0927 12:50:13.456155 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:13.456171 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:13.456369 (+ 198us) server_negotiation.cc:407] Connection header received
0927 12:50:13.456750 (+ 381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:13.456754 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:13.456919 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:13.457065 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:50:13.457259 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:13.458367 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:13.460093 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:13.460915 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:13.461019 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:13.461149 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:13.461905 (+ 756us) server_negotiation.cc:299] Negotiation successful
0927 12:50:13.461999 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":108,"threads_started":1}
I20260927 12:50:13.463738 19990 tablet_service.cc:1467] Processing CreateTablet for tablet 1844fbbd86924fc3a15a69e03e0b0118 (DEFAULT_TABLE table=test-table [id=6d3f43a623e34403b0521c27a904d11a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:50:13.464677 19990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1844fbbd86924fc3a15a69e03e0b0118. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:50:13.464915 19722 tablet_service.cc:1467] Processing CreateTablet for tablet 1844fbbd86924fc3a15a69e03e0b0118 (DEFAULT_TABLE table=test-table [id=6d3f43a623e34403b0521c27a904d11a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:50:13.465832 19722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1844fbbd86924fc3a15a69e03e0b0118. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:50:13.468914 19856 tablet_service.cc:1467] Processing CreateTablet for tablet 1844fbbd86924fc3a15a69e03e0b0118 (DEFAULT_TABLE table=test-table [id=6d3f43a623e34403b0521c27a904d11a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:50:13.469950 19856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1844fbbd86924fc3a15a69e03e0b0118. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:50:13.471791 20073 tablet_bootstrap.cc:492] T 1844fbbd86924fc3a15a69e03e0b0118 P 203447288e9c45bc92259702580fd2af: Bootstrap starting.
I20260927 12:50:13.473196 20074 tablet_bootstrap.cc:492] T 1844fbbd86924fc3a15a69e03e0b0118 P ff5147bcb6c84c5bae8939f8a213b5b4: Bootstrap starting.
I20260927 12:50:13.473934 20073 tablet_bootstrap.cc:654] T 1844fbbd86924fc3a15a69e03e0b0118 P 203447288e9c45bc92259702580fd2af: Neither blocks nor log segments found. Creating new log.
I20260927 12:50:13.475049 20073 log.cc:824] T 1844fbbd86924fc3a15a69e03e0b0118 P 203447288e9c45bc92259702580fd2af: Log is configured to *not* fsync() on all Append() calls
I20260927 12:50:13.475248 20074 tablet_bootstrap.cc:654] T 1844fbbd86924fc3a15a69e03e0b0118 P ff5147bcb6c84c5bae8939f8a213b5b4: Neither blocks nor log segments found. Creating new log.
I20260927 12:50:13.476055 20074 log.cc:824] T 1844fbbd86924fc3a15a69e03e0b0118 P ff5147bcb6c84c5bae8939f8a213b5b4: Log is configured to *not* fsync() on all Append() calls
I20260927 12:50:13.477262 20073 tablet_bootstrap.cc:492] T 1844fbbd86924fc3a15a69e03e0b0118 P 203447288e9c45bc92259702580fd2af: No bootstrap required, opened a new log
I20260927 12:50:13.477455 20073 ts_tablet_manager.cc:1397] T 1844fbbd86924fc3a15a69e03e0b0118 P 203447288e9c45bc92259702580fd2af: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 12:50:13.477874 20074 tablet_bootstrap.cc:492] T 1844fbbd86924fc3a15a69e03e0b0118 P ff5147bcb6c84c5bae8939f8a213b5b4: No bootstrap required, opened a new log
I20260927 12:50:13.478071 20074 ts_tablet_manager.cc:1397] T 1844fbbd86924fc3a15a69e03e0b0118 P ff5147bcb6c84c5bae8939f8a213b5b4: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260927 12:50:13.478106 20077 tablet_bootstrap.cc:492] T 1844fbbd86924fc3a15a69e03e0b0118 P 2c59ae273d4743778da9457d9c77a599: Bootstrap starting.
I20260927 12:50:13.480281 20077 tablet_bootstrap.cc:654] T 1844fbbd86924fc3a15a69e03e0b0118 P 2c59ae273d4743778da9457d9c77a599: Neither blocks nor log segments found. Creating new log.
I20260927 12:50:13.480669 20073 raft_consensus.cc:348] T 1844fbbd86924fc3a15a69e03e0b0118 P 203447288e9c45bc92259702580fd2af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "203447288e9c45bc92259702580fd2af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39875 } } peers { permanent_uuid: "ff5147bcb6c84c5bae8939f8a213b5b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36119 } } peers { permanent_uuid: "2c59ae273d4743778da9457d9c77a599" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46095 } }
I20260927 12:50:13.480942 20073 raft_consensus.cc:374] T 1844fbbd86924fc3a15a69e03e0b0118 P 203447288e9c45bc92259702580fd2af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:50:13.480995 20073 raft_consensus.cc:729] T 1844fbbd86924fc3a15a69e03e0b0118 P 203447288e9c45bc92259702580fd2af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 203447288e9c45bc92259702580fd2af, State: Initialized, Role: FOLLOWER
I20260927 12:50:13.481422 20073 consensus_queue.cc:260] T 1844fbbd86924fc3a15a69e03e0b0118 P 203447288e9c45bc92259702580fd2af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "203447288e9c45bc92259702580fd2af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39875 } } peers { permanent_uuid: "ff5147bcb6c84c5bae8939f8a213b5b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36119 } } peers { permanent_uuid: "2c59ae273d4743778da9457d9c77a599" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46095 } }
I20260927 12:50:13.481489 20077 log.cc:824] T 1844fbbd86924fc3a15a69e03e0b0118 P 2c59ae273d4743778da9457d9c77a599: Log is configured to *not* fsync() on all Append() calls
I20260927 12:50:13.482463 20073 ts_tablet_manager.cc:1428] T 1844fbbd86924fc3a15a69e03e0b0118 P 203447288e9c45bc92259702580fd2af: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 12:50:13.482350 20074 raft_consensus.cc:348] T 1844fbbd86924fc3a15a69e03e0b0118 P ff5147bcb6c84c5bae8939f8a213b5b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "203447288e9c45bc92259702580fd2af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39875 } } peers { permanent_uuid: "ff5147bcb6c84c5bae8939f8a213b5b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36119 } } peers { permanent_uuid: "2c59ae273d4743778da9457d9c77a599" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46095 } }
I20260927 12:50:13.482599 20074 raft_consensus.cc:374] T 1844fbbd86924fc3a15a69e03e0b0118 P ff5147bcb6c84c5bae8939f8a213b5b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:50:13.482664 20074 raft_consensus.cc:729] T 1844fbbd86924fc3a15a69e03e0b0118 P ff5147bcb6c84c5bae8939f8a213b5b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5147bcb6c84c5bae8939f8a213b5b4, State: Initialized, Role: FOLLOWER
I20260927 12:50:13.483000 20056 heartbeater.cc:503] Master 127.0.98.62:35029 was elected leader, sending a full tablet report...
I20260927 12:50:13.483094 20074 consensus_queue.cc:260] T 1844fbbd86924fc3a15a69e03e0b0118 P ff5147bcb6c84c5bae8939f8a213b5b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "203447288e9c45bc92259702580fd2af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39875 } } peers { permanent_uuid: "ff5147bcb6c84c5bae8939f8a213b5b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36119 } } peers { permanent_uuid: "2c59ae273d4743778da9457d9c77a599" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46095 } }
I20260927 12:50:13.483840 20077 tablet_bootstrap.cc:492] T 1844fbbd86924fc3a15a69e03e0b0118 P 2c59ae273d4743778da9457d9c77a599: No bootstrap required, opened a new log
I20260927 12:50:13.484025 20077 ts_tablet_manager.cc:1397] T 1844fbbd86924fc3a15a69e03e0b0118 P 2c59ae273d4743778da9457d9c77a599: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 12:50:13.484023 20074 ts_tablet_manager.cc:1428] T 1844fbbd86924fc3a15a69e03e0b0118 P ff5147bcb6c84c5bae8939f8a213b5b4: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 12:50:13.484336 19788 heartbeater.cc:503] Master 127.0.98.62:35029 was elected leader, sending a full tablet report...
I20260927 12:50:13.488083 20077 raft_consensus.cc:348] T 1844fbbd86924fc3a15a69e03e0b0118 P 2c59ae273d4743778da9457d9c77a599 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "203447288e9c45bc92259702580fd2af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39875 } } peers { permanent_uuid: "ff5147bcb6c84c5bae8939f8a213b5b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36119 } } peers { permanent_uuid: "2c59ae273d4743778da9457d9c77a599" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46095 } }
I20260927 12:50:13.488324 20077 raft_consensus.cc:374] T 1844fbbd86924fc3a15a69e03e0b0118 P 2c59ae273d4743778da9457d9c77a599 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:50:13.488358 20077 raft_consensus.cc:729] T 1844fbbd86924fc3a15a69e03e0b0118 P 2c59ae273d4743778da9457d9c77a599 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c59ae273d4743778da9457d9c77a599, State: Initialized, Role: FOLLOWER
I20260927 12:50:13.488700 20077 consensus_queue.cc:260] T 1844fbbd86924fc3a15a69e03e0b0118 P 2c59ae273d4743778da9457d9c77a599 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "203447288e9c45bc92259702580fd2af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39875 } } peers { permanent_uuid: "ff5147bcb6c84c5bae8939f8a213b5b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36119 } } peers { permanent_uuid: "2c59ae273d4743778da9457d9c77a599" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46095 } }
I20260927 12:50:13.489275 19922 heartbeater.cc:503] Master 127.0.98.62:35029 was elected leader, sending a full tablet report...
I20260927 12:50:13.489506 20077 ts_tablet_manager.cc:1428] T 1844fbbd86924fc3a15a69e03e0b0118 P 2c59ae273d4743778da9457d9c77a599: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 12:50:13.574370 20081 raft_consensus.cc:482] T 1844fbbd86924fc3a15a69e03e0b0118 P 2c59ae273d4743778da9457d9c77a599 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:50:13.574579 20081 raft_consensus.cc:504] T 1844fbbd86924fc3a15a69e03e0b0118 P 2c59ae273d4743778da9457d9c77a599 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "203447288e9c45bc92259702580fd2af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39875 } } peers { permanent_uuid: "ff5147bcb6c84c5bae8939f8a213b5b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36119 } } peers { permanent_uuid: "2c59ae273d4743778da9457d9c77a599" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46095 } }
I20260927 12:50:13.575786 20081 leader_election.cc:288] T 1844fbbd86924fc3a15a69e03e0b0118 P 2c59ae273d4743778da9457d9c77a599 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 203447288e9c45bc92259702580fd2af (127.0.98.3:39875), ff5147bcb6c84c5bae8939f8a213b5b4 (127.0.98.1:36119)
I20260927 12:50:13.579886 19924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:13.576142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39875 (local address 127.0.98.2:34769)
0927 12:50:13.576257 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:13.576273 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:50:13.576356 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:50:13.576601 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:50:13.576604 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:50:13.576625 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:50:13.576746 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:13.576753 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:13.577625 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:13.577628 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:13.578631 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:13.578640 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:13.579464 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:13.579468 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:13.579576 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:13.579601 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 12:50:13.579675 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 12:50:13.579744 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 12:50:13.579895 20068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:13.576245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34769 (local address 127.0.98.3:39875)
0927 12:50:13.576363 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:13.576368 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:13.576384 (+ 16us) server_negotiation.cc:407] Connection header received
0927 12:50:13.576436 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:13.576439 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:13.576488 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:13.576578 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:50:13.576866 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:13.577525 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:13.578762 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:13.579353 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:13.579441 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:13.579496 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:13.579732 (+ 236us) server_negotiation.cc:299] Negotiation successful
0927 12:50:13.579798 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 12:50:13.580555 20082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:13.576381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36119 (local address 127.0.98.2:58805)
0927 12:50:13.576708 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:13.576722 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:50:13.576805 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:50:13.577132 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:50:13.577135 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:50:13.577151 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:50:13.577268 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:13.577274 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:13.578246 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:13.578248 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:13.579259 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:13.579269 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:13.580169 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:13.580173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:13.580268 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:13.580291 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 12:50:13.580367 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 12:50:13.580429 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":113,"threads_started":1}
I20260927 12:50:13.580564 20072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:13.576781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58805 (local address 127.0.98.1:36119)
0927 12:50:13.576910 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:13.576915 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:13.576925 (+ 10us) server_negotiation.cc:407] Connection header received
0927 12:50:13.576977 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:13.576979 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:13.577014 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:13.577065 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:50:13.577374 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:13.578126 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:13.579393 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:13.580009 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:13.580104 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:13.580162 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:13.580389 (+ 227us) server_negotiation.cc:299] Negotiation successful
0927 12:50:13.580460 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 12:50:13.580566 20010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1844fbbd86924fc3a15a69e03e0b0118" candidate_uuid: "2c59ae273d4743778da9457d9c77a599" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "203447288e9c45bc92259702580fd2af" is_pre_election: true
I20260927 12:50:13.580896 20010 raft_consensus.cc:2454] T 1844fbbd86924fc3a15a69e03e0b0118 P 203447288e9c45bc92259702580fd2af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c59ae273d4743778da9457d9c77a599 in term 0.
I20260927 12:50:13.581218 19742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1844fbbd86924fc3a15a69e03e0b0118" candidate_uuid: "2c59ae273d4743778da9457d9c77a599" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5147bcb6c84c5bae8939f8a213b5b4" is_pre_election: true
I20260927 12:50:13.581410 19810 leader_election.cc:302] T 1844fbbd86924fc3a15a69e03e0b0118 P 2c59ae273d4743778da9457d9c77a599 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 203447288e9c45bc92259702580fd2af, 2c59ae273d4743778da9457d9c77a599; no voters:
I20260927 12:50:13.581470 19742 raft_consensus.cc:2454] T 1844fbbd86924fc3a15a69e03e0b0118 P ff5147bcb6c84c5bae8939f8a213b5b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c59ae273d4743778da9457d9c77a599 in term 0.
I20260927 12:50:13.581676 20081 raft_consensus.cc:2781] T 1844fbbd86924fc3a15a69e03e0b0118 P 2c59ae273d4743778da9457d9c77a599 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:50:13.581772 20081 raft_consensus.cc:482] T 1844fbbd86924fc3a15a69e03e0b0118 P 2c59ae273d4743778da9457d9c77a599 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:50:13.581821 20081 raft_consensus.cc:3037] T 1844fbbd86924fc3a15a69e03e0b0118 P 2c59ae273d4743778da9457d9c77a599 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:50:13.582847 20081 raft_consensus.cc:504] T 1844fbbd86924fc3a15a69e03e0b0118 P 2c59ae273d4743778da9457d9c77a599 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "203447288e9c45bc92259702580fd2af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39875 } } peers { permanent_uuid: "ff5147bcb6c84c5bae8939f8a213b5b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36119 } } peers { permanent_uuid: "2c59ae273d4743778da9457d9c77a599" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46095 } }
I20260927 12:50:13.583177 20081 leader_election.cc:288] T 1844fbbd86924fc3a15a69e03e0b0118 P 2c59ae273d4743778da9457d9c77a599 [CANDIDATE]: Term 1 election: Requested vote from peers 203447288e9c45bc92259702580fd2af (127.0.98.3:39875), ff5147bcb6c84c5bae8939f8a213b5b4 (127.0.98.1:36119)
I20260927 12:50:13.583500 20010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1844fbbd86924fc3a15a69e03e0b0118" candidate_uuid: "2c59ae273d4743778da9457d9c77a599" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "203447288e9c45bc92259702580fd2af"
I20260927 12:50:13.583563 19742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1844fbbd86924fc3a15a69e03e0b0118" candidate_uuid: "2c59ae273d4743778da9457d9c77a599" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff5147bcb6c84c5bae8939f8a213b5b4"
I20260927 12:50:13.583611 20010 raft_consensus.cc:3037] T 1844fbbd86924fc3a15a69e03e0b0118 P 203447288e9c45bc92259702580fd2af [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:50:13.583693 19742 raft_consensus.cc:3037] T 1844fbbd86924fc3a15a69e03e0b0118 P ff5147bcb6c84c5bae8939f8a213b5b4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:50:13.584728 20010 raft_consensus.cc:2454] T 1844fbbd86924fc3a15a69e03e0b0118 P 203447288e9c45bc92259702580fd2af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c59ae273d4743778da9457d9c77a599 in term 1.
I20260927 12:50:13.584792 19742 raft_consensus.cc:2454] T 1844fbbd86924fc3a15a69e03e0b0118 P ff5147bcb6c84c5bae8939f8a213b5b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c59ae273d4743778da9457d9c77a599 in term 1.
I20260927 12:50:13.585018 19810 leader_election.cc:302] T 1844fbbd86924fc3a15a69e03e0b0118 P 2c59ae273d4743778da9457d9c77a599 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 203447288e9c45bc92259702580fd2af, 2c59ae273d4743778da9457d9c77a599; no voters:
I20260927 12:50:13.585176 20081 raft_consensus.cc:2781] T 1844fbbd86924fc3a15a69e03e0b0118 P 2c59ae273d4743778da9457d9c77a599 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:50:13.585384 20081 raft_consensus.cc:686] T 1844fbbd86924fc3a15a69e03e0b0118 P 2c59ae273d4743778da9457d9c77a599 [term 1 LEADER]: Becoming Leader. State: Replica: 2c59ae273d4743778da9457d9c77a599, State: Running, Role: LEADER
I20260927 12:50:13.585767 20081 consensus_queue.cc:237] T 1844fbbd86924fc3a15a69e03e0b0118 P 2c59ae273d4743778da9457d9c77a599 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "203447288e9c45bc92259702580fd2af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39875 } } peers { permanent_uuid: "ff5147bcb6c84c5bae8939f8a213b5b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36119 } } peers { permanent_uuid: "2c59ae273d4743778da9457d9c77a599" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46095 } }
I20260927 12:50:13.589165 19597 catalog_manager.cc:5495] T 1844fbbd86924fc3a15a69e03e0b0118 P 2c59ae273d4743778da9457d9c77a599 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c59ae273d4743778da9457d9c77a599 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "2c59ae273d4743778da9457d9c77a599" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "203447288e9c45bc92259702580fd2af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff5147bcb6c84c5bae8939f8a213b5b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c59ae273d4743778da9457d9c77a599" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46095 } health_report { overall_health: HEALTHY } } }
I20260927 12:50:13.614540 20070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:13.610848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46500 (local address 127.0.98.2:46095)
0927 12:50:13.610970 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:13.610974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:13.610987 (+ 13us) server_negotiation.cc:407] Connection header received
0927 12:50:13.611094 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:13.611096 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:13.611152 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:13.611276 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:50:13.611555 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:13.612082 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:13.613016 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:13.613226 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:13.613293 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:13.613586 (+ 293us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:50:13.614077 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:50:13.614167 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:13.614355 (+ 188us) server_negotiation.cc:299] Negotiation successful
0927 12:50:13.614418 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 12:50:13.621207 20010 raft_consensus.cc:1261] T 1844fbbd86924fc3a15a69e03e0b0118 P 203447288e9c45bc92259702580fd2af [term 1 FOLLOWER]: Refusing update from remote peer 2c59ae273d4743778da9457d9c77a599: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:50:13.621197 19742 raft_consensus.cc:1261] T 1844fbbd86924fc3a15a69e03e0b0118 P ff5147bcb6c84c5bae8939f8a213b5b4 [term 1 FOLLOWER]: Refusing update from remote peer 2c59ae273d4743778da9457d9c77a599: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:50:13.621908 20081 consensus_queue.cc:1035] T 1844fbbd86924fc3a15a69e03e0b0118 P 2c59ae273d4743778da9457d9c77a599 [LEADER]: Connected to new peer: Peer: permanent_uuid: "203447288e9c45bc92259702580fd2af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:50:13.622157 20083 consensus_queue.cc:1035] T 1844fbbd86924fc3a15a69e03e0b0118 P 2c59ae273d4743778da9457d9c77a599 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff5147bcb6c84c5bae8939f8a213b5b4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:50:13.638548 19597 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:50:13.638798 19597 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:50:13.641347 19597 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 99a6bc1e2fda4f17ab8b6344622708e7: Sending DeleteTablet for 3 replicas of tablet 1844fbbd86924fc3a15a69e03e0b0118
I20260927 12:50:13.642285 19856 tablet_service.cc:1514] Processing DeleteTablet for tablet 1844fbbd86924fc3a15a69e03e0b0118 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:50:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46484
I20260927 12:50:13.642417 19722 tablet_service.cc:1514] Processing DeleteTablet for tablet 1844fbbd86924fc3a15a69e03e0b0118 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:50:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52066
I20260927 12:50:13.643203 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 19659
I20260927 12:50:13.643187 19990 tablet_service.cc:1514] Processing DeleteTablet for tablet 1844fbbd86924fc3a15a69e03e0b0118 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:50:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57346
I20260927 12:50:13.644230 20098 tablet_replica.cc:330] T 1844fbbd86924fc3a15a69e03e0b0118 P 203447288e9c45bc92259702580fd2af: stopping tablet replica
I20260927 12:50:13.644587 20098 raft_consensus.cc:2229] T 1844fbbd86924fc3a15a69e03e0b0118 P 203447288e9c45bc92259702580fd2af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:50:13.644933 20098 raft_consensus.cc:2258] T 1844fbbd86924fc3a15a69e03e0b0118 P 203447288e9c45bc92259702580fd2af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:50:13.648096 20096 tablet_replica.cc:330] T 1844fbbd86924fc3a15a69e03e0b0118 P 2c59ae273d4743778da9457d9c77a599: stopping tablet replica
I20260927 12:50:13.648450 20096 raft_consensus.cc:2229] T 1844fbbd86924fc3a15a69e03e0b0118 P 2c59ae273d4743778da9457d9c77a599 [term 1 LEADER]: Raft consensus shutting down.
I20260927 12:50:13.648725 20098 ts_tablet_manager.cc:1905] T 1844fbbd86924fc3a15a69e03e0b0118 P 203447288e9c45bc92259702580fd2af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:50:13.648835 20096 raft_consensus.cc:2258] T 1844fbbd86924fc3a15a69e03e0b0118 P 2c59ae273d4743778da9457d9c77a599 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:50:13.650395 20096 ts_tablet_manager.cc:1905] T 1844fbbd86924fc3a15a69e03e0b0118 P 2c59ae273d4743778da9457d9c77a599: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:50:13.653018 20096 ts_tablet_manager.cc:1918] T 1844fbbd86924fc3a15a69e03e0b0118 P 2c59ae273d4743778da9457d9c77a599: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:50:13.653115 20096 log.cc:1196] T 1844fbbd86924fc3a15a69e03e0b0118 P 2c59ae273d4743778da9457d9c77a599: Deleting WAL directory at /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/minicluster-data/ts-1/wal/wals/1844fbbd86924fc3a15a69e03e0b0118
I20260927 12:50:13.653399 20096 ts_tablet_manager.cc:1939] T 1844fbbd86924fc3a15a69e03e0b0118 P 2c59ae273d4743778da9457d9c77a599: Deleting consensus metadata
I20260927 12:50:13.654584 19583 catalog_manager.cc:4841] TS 2c59ae273d4743778da9457d9c77a599 (127.0.98.2:46095): tablet 1844fbbd86924fc3a15a69e03e0b0118 (table test-table [id=6d3f43a623e34403b0521c27a904d11a]) successfully deleted
W20260927 12:50:13.654603 19582 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36119 (error 108)
I20260927 12:50:13.656088 20071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:13.655754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36119 (local address 127.0.0.1:52068)
0927 12:50:13.655941 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:13.656004 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
W20260927 12:50:13.656373 19582 catalog_manager.cc:4584] TS ff5147bcb6c84c5bae8939f8a213b5b4 (127.0.98.1:36119): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1844fbbd86924fc3a15a69e03e0b0118: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36119: connect: Connection refused (error 111)
I20260927 12:50:13.656495 20098 ts_tablet_manager.cc:1918] T 1844fbbd86924fc3a15a69e03e0b0118 P 203447288e9c45bc92259702580fd2af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:50:13.656587 20098 log.cc:1196] T 1844fbbd86924fc3a15a69e03e0b0118 P 203447288e9c45bc92259702580fd2af: Deleting WAL directory at /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790513161160093-392-0/minicluster-data/ts-2/wal/wals/1844fbbd86924fc3a15a69e03e0b0118
I20260927 12:50:13.656770 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 19793
I20260927 12:50:13.656878 20098 ts_tablet_manager.cc:1939] T 1844fbbd86924fc3a15a69e03e0b0118 P 203447288e9c45bc92259702580fd2af: Deleting consensus metadata
I20260927 12:50:13.658730 19582 catalog_manager.cc:4841] TS 203447288e9c45bc92259702580fd2af (127.0.98.3:39875): tablet 1844fbbd86924fc3a15a69e03e0b0118 (table test-table [id=6d3f43a623e34403b0521c27a904d11a]) successfully deleted
I20260927 12:50:13.664034 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 19927
I20260927 12:50:13.669274 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 19565
2026-09-27T12:50:13Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2273 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-27T12:50:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:50:13Z Disabled control of system clock
I20260927 12:50:13.698091 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790513161160093-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790513161160093-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790513161160093-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41761
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41761
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 12:50:13.809783 20103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:50:13.810184 20103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:50:13.810227 20103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:50:13.814464 20103 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:50:13.814540 20103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:50:13.814554 20103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:50:13.814571 20103 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:50:13.814585 20103 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:50:13.819592 20103 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790513161160093-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790513161160093-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41761
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790513161160093-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:50:13.820735 20103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:50:13.822173 20103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:50:13.831233 20108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:50:13.831231 20111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:50:13.831311 20109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:50:13.831765 20103 server_base.cc:1034] running on GCE node
I20260927 12:50:13.832350 20103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:50:13.833593 20103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:50:13.834829 20103 hybrid_clock.cc:648] HybridClock initialized: now 1790513413834802 us; error 40 us; skew 500 ppm
I20260927 12:50:13.836807 20103 webserver.cc:460] Webserver started at http://127.0.98.62:41845/ using document root <none> and password file <none>
I20260927 12:50:13.837371 20103 fs_manager.cc:359] Metadata directory not provided
I20260927 12:50:13.837435 20103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:50:13.837625 20103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:50:13.839789 20103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790513161160093-392-0/minicluster-data/master-0/data/instance:
uuid: "c92c03b0e4bc403097eecbf37fb1f42e"
format_stamp: "Formatted at 2026-09-27 12:50:13 on dist-test-slave-lkn4"
server_key: "09247c17021eafa457e2f976a192d81d"
server_key_iv: "7c0f5b8015824fb4eb0cbd1c766bb69e"
server_key_version: "encryptionkey@0"
I20260927 12:50:13.840354 20103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790513161160093-392-0/minicluster-data/master-0/wal/instance:
uuid: "c92c03b0e4bc403097eecbf37fb1f42e"
format_stamp: "Formatted at 2026-09-27 12:50:13 on dist-test-slave-lkn4"
server_key: "09247c17021eafa457e2f976a192d81d"
server_key_iv: "7c0f5b8015824fb4eb0cbd1c766bb69e"
server_key_version: "encryptionkey@0"
I20260927 12:50:13.844107 20103 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260927 12:50:13.846526 20117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:13.847744 20103 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:50:13.847863 20103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790513161160093-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790513161160093-392-0/minicluster-data/master-0/wal
uuid: "c92c03b0e4bc403097eecbf37fb1f42e"
format_stamp: "Formatted at 2026-09-27 12:50:13 on dist-test-slave-lkn4"
server_key: "09247c17021eafa457e2f976a192d81d"
server_key_iv: "7c0f5b8015824fb4eb0cbd1c766bb69e"
server_key_version: "encryptionkey@0"
I20260927 12:50:13.847959 20103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790513161160093-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790513161160093-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790513161160093-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:50:13.864364 20103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:50:13.865072 20103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:50:13.865245 20103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:50:13.872113 20103 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41761
I20260927 12:50:13.872159 20169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41761 every 8 connection(s)
I20260927 12:50:13.873214 20103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:50:13.873790 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 20103
I20260927 12:50:13.873910 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790513161160093-392-0/minicluster-data/master-0/wal/instance
I20260927 12:50:13.874224 392 external_mini_cluster.cc:1428] Setting key 230e563d2834858e7dc8d35c8bb8f237
I20260927 12:50:13.876755 20170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:50:13.882519 20170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c92c03b0e4bc403097eecbf37fb1f42e: Bootstrap starting.
I20260927 12:50:13.883494 20173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:13.875588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38436 (local address 127.0.98.62:41761)
0927 12:50:13.876296 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:13.876317 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:13.876362 (+ 45us) server_negotiation.cc:407] Connection header received
0927 12:50:13.877073 (+ 711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:13.877093 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:13.877504 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:13.877845 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:13.878325 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:13.879291 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:13.880181 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:13.880685 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:13.880836 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:13.881672 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:13.881677 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:13.881691 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:50:13.882135 (+ 444us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:13.882199 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:50:13.882220 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:13.882306 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:13.882441 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 12:50:13.882723 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":109,"threads_started":1}
I20260927 12:50:13.884964 20170 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c92c03b0e4bc403097eecbf37fb1f42e: Neither blocks nor log segments found. Creating new log.
I20260927 12:50:13.885764 20170 log.cc:824] T 00000000000000000000000000000000 P c92c03b0e4bc403097eecbf37fb1f42e: Log is configured to *not* fsync() on all Append() calls
I20260927 12:50:13.888092 20170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c92c03b0e4bc403097eecbf37fb1f42e: No bootstrap required, opened a new log
I20260927 12:50:13.891795 20170 raft_consensus.cc:348] T 00000000000000000000000000000000 P c92c03b0e4bc403097eecbf37fb1f42e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c92c03b0e4bc403097eecbf37fb1f42e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41761 } }
I20260927 12:50:13.892038 20170 raft_consensus.cc:374] T 00000000000000000000000000000000 P c92c03b0e4bc403097eecbf37fb1f42e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:50:13.892099 20170 raft_consensus.cc:729] T 00000000000000000000000000000000 P c92c03b0e4bc403097eecbf37fb1f42e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92c03b0e4bc403097eecbf37fb1f42e, State: Initialized, Role: FOLLOWER
I20260927 12:50:13.892537 20170 consensus_queue.cc:260] T 00000000000000000000000000000000 P c92c03b0e4bc403097eecbf37fb1f42e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c92c03b0e4bc403097eecbf37fb1f42e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41761 } }
I20260927 12:50:13.892673 20170 raft_consensus.cc:388] T 00000000000000000000000000000000 P c92c03b0e4bc403097eecbf37fb1f42e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:50:13.892736 20170 raft_consensus.cc:482] T 00000000000000000000000000000000 P c92c03b0e4bc403097eecbf37fb1f42e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:50:13.892835 20170 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c92c03b0e4bc403097eecbf37fb1f42e [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:50:13.893746 20170 raft_consensus.cc:504] T 00000000000000000000000000000000 P c92c03b0e4bc403097eecbf37fb1f42e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c92c03b0e4bc403097eecbf37fb1f42e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41761 } }
I20260927 12:50:13.894121 20170 leader_election.cc:302] T 00000000000000000000000000000000 P c92c03b0e4bc403097eecbf37fb1f42e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c92c03b0e4bc403097eecbf37fb1f42e; no voters:
I20260927 12:50:13.894440 20170 leader_election.cc:288] T 00000000000000000000000000000000 P c92c03b0e4bc403097eecbf37fb1f42e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:50:13.894554 20175 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c92c03b0e4bc403097eecbf37fb1f42e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:50:13.894812 20175 raft_consensus.cc:686] T 00000000000000000000000000000000 P c92c03b0e4bc403097eecbf37fb1f42e [term 1 LEADER]: Becoming Leader. State: Replica: c92c03b0e4bc403097eecbf37fb1f42e, State: Running, Role: LEADER
I20260927 12:50:13.895153 20175 consensus_queue.cc:237] T 00000000000000000000000000000000 P c92c03b0e4bc403097eecbf37fb1f42e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c92c03b0e4bc403097eecbf37fb1f42e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41761 } }
I20260927 12:50:13.895721 20170 sys_catalog.cc:564] T 00000000000000000000000000000000 P c92c03b0e4bc403097eecbf37fb1f42e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:50:13.896706 20176 sys_catalog.cc:455] T 00000000000000000000000000000000 P c92c03b0e4bc403097eecbf37fb1f42e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c92c03b0e4bc403097eecbf37fb1f42e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c92c03b0e4bc403097eecbf37fb1f42e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41761 } } }
I20260927 12:50:13.896835 20176 sys_catalog.cc:458] T 00000000000000000000000000000000 P c92c03b0e4bc403097eecbf37fb1f42e [sys.catalog]: This master's current role is: LEADER
I20260927 12:50:13.897106 20177 sys_catalog.cc:455] T 00000000000000000000000000000000 P c92c03b0e4bc403097eecbf37fb1f42e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c92c03b0e4bc403097eecbf37fb1f42e. Latest consensus state: current_term: 1 leader_uuid: "c92c03b0e4bc403097eecbf37fb1f42e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c92c03b0e4bc403097eecbf37fb1f42e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41761 } } }
I20260927 12:50:13.897203 20177 sys_catalog.cc:458] T 00000000000000000000000000000000 P c92c03b0e4bc403097eecbf37fb1f42e [sys.catalog]: This master's current role is: LEADER
I20260927 12:50:13.897320 20184 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:50:13.900314 20184 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:50:13.905620 20184 catalog_manager.cc:1347] Generated new cluster ID: 5c6ca0ddb8424b0da70c1d8373105165
I20260927 12:50:13.905699 20184 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:50:13.914454 20184 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:50:13.915719 20184 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:50:13.928180 20184 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c92c03b0e4bc403097eecbf37fb1f42e: Generated new TSK 0
I20260927 12:50:13.929107 20184 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:50:13.931613 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790513161160093-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790513161160093-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790513161160093-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41761
--builtin_ntp_servers=127.0.98.20:36433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 12:50:14.043989 20194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:50:14.044275 20194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:50:14.044313 20194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:50:14.048447 20194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:50:14.048512 20194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:50:14.048569 20194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:50:14.053620 20194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790513161160093-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790513161160093-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790513161160093-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:50:14.054854 20194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:50:14.056313 20194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:50:14.066430 20200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:50:14.066430 20202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:50:14.066432 20199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:50:14.066859 20194 server_base.cc:1034] running on GCE node
I20260927 12:50:14.067463 20194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:50:14.068146 20194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:50:14.069319 20194 hybrid_clock.cc:648] HybridClock initialized: now 1790513414069284 us; error 48 us; skew 500 ppm
I20260927 12:50:14.071518 20194 webserver.cc:460] Webserver started at http://127.0.98.1:40023/ using document root <none> and password file <none>
I20260927 12:50:14.072130 20194 fs_manager.cc:359] Metadata directory not provided
I20260927 12:50:14.072196 20194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:50:14.072414 20194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:50:14.074661 20194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790513161160093-392-0/minicluster-data/ts-0/data/instance:
uuid: "4621ea9b61a846a298d45faeac40009c"
format_stamp: "Formatted at 2026-09-27 12:50:14 on dist-test-slave-lkn4"
server_key: "4f3bdca71ec1d768d11a0c4f93b36bd9"
server_key_iv: "469da1ad519bf90967eaee0af786eda6"
server_key_version: "encryptionkey@0"
I20260927 12:50:14.075258 20194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790513161160093-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4621ea9b61a846a298d45faeac40009c"
format_stamp: "Formatted at 2026-09-27 12:50:14 on dist-test-slave-lkn4"
server_key: "4f3bdca71ec1d768d11a0c4f93b36bd9"
server_key_iv: "469da1ad519bf90967eaee0af786eda6"
server_key_version: "encryptionkey@0"
I20260927 12:50:14.079356 20194 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260927 12:50:14.082173 20208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:14.083602 20194 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 12:50:14.083751 20194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790513161160093-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790513161160093-392-0/minicluster-data/ts-0/wal
uuid: "4621ea9b61a846a298d45faeac40009c"
format_stamp: "Formatted at 2026-09-27 12:50:14 on dist-test-slave-lkn4"
server_key: "4f3bdca71ec1d768d11a0c4f93b36bd9"
server_key_iv: "469da1ad519bf90967eaee0af786eda6"
server_key_version: "encryptionkey@0"
I20260927 12:50:14.083869 20194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790513161160093-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790513161160093-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790513161160093-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:50:14.102620 20194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:50:14.103382 20194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:50:14.103569 20194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:50:14.104189 20194 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:50:14.105208 20194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:50:14.105257 20194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:14.105309 20194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:50:14.105338 20194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:14.113821 20194 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35649
I20260927 12:50:14.113859 20321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35649 every 8 connection(s)
I20260927 12:50:14.114787 20194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790513161160093-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:50:14.118023 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 20194
I20260927 12:50:14.118178 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790513161160093-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:50:14.118510 392 external_mini_cluster.cc:1428] Setting key 6511f68d34ebfd42fb302665b99941f3
I20260927 12:50:14.120584 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790513161160093-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790513161160093-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790513161160093-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41761
--builtin_ntp_servers=127.0.98.20:36433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 12:50:14.125049 20173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:14.116729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56423 (local address 127.0.98.62:41761)
0927 12:50:14.116891 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:14.116895 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:14.118166 (+ 1271us) server_negotiation.cc:407] Connection header received
0927 12:50:14.120039 (+ 1873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:14.120042 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:14.120092 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:14.120171 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:14.121417 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:14.122008 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:14.123214 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:14.123436 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:14.123524 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:14.124574 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:14.124577 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:14.124579 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:50:14.124592 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:14.124610 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:50:14.124618 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:14.124674 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:14.124816 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 12:50:14.124891 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 12:50:14.125918 20324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:14.117196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41761 (local address 127.0.98.1:56423)
0927 12:50:14.117966 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:14.118027 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 12:50:14.119637 (+ 1610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:50:14.120404 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:50:14.120413 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:50:14.120539 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:50:14.121255 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:14.121273 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:14.122148 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:14.122151 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:14.123097 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:14.123106 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:14.123578 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:14.123582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:14.123752 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:14.124320 (+ 568us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 12:50:14.124322 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:50:14.124364 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 12:50:14.124431 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:50:14.124684 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:50:14.124693 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:50:14.124699 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 12:50:14.124811 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 12:50:14.125101 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":119,"threads_started":1}
I20260927 12:50:14.127265 20322 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41761
I20260927 12:50:14.127486 20322 heartbeater.cc:464] Registering TS with master...
I20260927 12:50:14.128077 20322 heartbeater.cc:511] Master 127.0.98.62:41761 requested a full tablet report, sending...
I20260927 12:50:14.129665 20134 ts_manager.cc:194] Registered new tserver with Master: 4621ea9b61a846a298d45faeac40009c (127.0.98.1:35649)
I20260927 12:50:14.130897 20134 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:56423
W20260927 12:50:14.229928 20325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:50:14.230310 20325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:50:14.230391 20325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:50:14.234622 20325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:50:14.234690 20325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:50:14.234771 20325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:50:14.239887 20325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790513161160093-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790513161160093-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790513161160093-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:50:14.241007 20325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:50:14.242455 20325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:50:14.252146 20330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:50:14.252146 20331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:50:14.252256 20333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:50:14.252483 20325 server_base.cc:1034] running on GCE node
I20260927 12:50:14.252992 20325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:50:14.253635 20325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:50:14.254802 20325 hybrid_clock.cc:648] HybridClock initialized: now 1790513414254781 us; error 31 us; skew 500 ppm
I20260927 12:50:14.256636 20325 webserver.cc:460] Webserver started at http://127.0.98.2:41155/ using document root <none> and password file <none>
I20260927 12:50:14.257251 20325 fs_manager.cc:359] Metadata directory not provided
I20260927 12:50:14.257315 20325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:50:14.257509 20325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:50:14.259584 20325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790513161160093-392-0/minicluster-data/ts-1/data/instance:
uuid: "eb2233ac568342c39c675fbff008197e"
format_stamp: "Formatted at 2026-09-27 12:50:14 on dist-test-slave-lkn4"
server_key: "af2e138cbf71ba77b7a9ec9175953ca6"
server_key_iv: "cd3dd106c3873a6cf11ff42165a25b22"
server_key_version: "encryptionkey@0"
I20260927 12:50:14.260115 20325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790513161160093-392-0/minicluster-data/ts-1/wal/instance:
uuid: "eb2233ac568342c39c675fbff008197e"
format_stamp: "Formatted at 2026-09-27 12:50:14 on dist-test-slave-lkn4"
server_key: "af2e138cbf71ba77b7a9ec9175953ca6"
server_key_iv: "cd3dd106c3873a6cf11ff42165a25b22"
server_key_version: "encryptionkey@0"
I20260927 12:50:14.264041 20325 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 12:50:14.266558 20339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:14.267978 20325 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:50:14.268117 20325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790513161160093-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790513161160093-392-0/minicluster-data/ts-1/wal
uuid: "eb2233ac568342c39c675fbff008197e"
format_stamp: "Formatted at 2026-09-27 12:50:14 on dist-test-slave-lkn4"
server_key: "af2e138cbf71ba77b7a9ec9175953ca6"
server_key_iv: "cd3dd106c3873a6cf11ff42165a25b22"
server_key_version: "encryptionkey@0"
I20260927 12:50:14.268273 20325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790513161160093-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790513161160093-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790513161160093-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:50:14.334167 20325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:50:14.334928 20325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:50:14.335109 20325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:50:14.335714 20325 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:50:14.336706 20325 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:50:14.336766 20325 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:14.336809 20325 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:50:14.336838 20325 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:14.345363 20325 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35515
I20260927 12:50:14.345393 20452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35515 every 8 connection(s)
I20260927 12:50:14.346359 20325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790513161160093-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:50:14.347694 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 20325
I20260927 12:50:14.347822 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790513161160093-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:50:14.348107 392 external_mini_cluster.cc:1428] Setting key 850439a6955b905d9d83c6bb5fbf168c
I20260927 12:50:14.349999 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790513161160093-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790513161160093-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790513161160093-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41761
--builtin_ntp_servers=127.0.98.20:36433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 12:50:14.356547 20173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:14.348314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37371 (local address 127.0.98.62:41761)
0927 12:50:14.348464 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:14.348469 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:14.349716 (+ 1247us) server_negotiation.cc:407] Connection header received
0927 12:50:14.351225 (+ 1509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:14.351229 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:14.351285 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:14.351357 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:14.352292 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:14.353051 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:14.354294 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:14.354636 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:14.354754 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:14.356032 (+ 1278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:14.356036 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:14.356039 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:50:14.356056 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:14.356082 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:50:14.356094 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:14.356175 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:14.356305 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 12:50:14.356382 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 12:50:14.357640 20455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:14.348843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41761 (local address 127.0.98.2:37371)
0927 12:50:14.349559 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:14.349611 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 12:50:14.351007 (+ 1396us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:50:14.351544 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:50:14.351552 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:50:14.351642 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:50:14.352153 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:14.352165 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:14.353188 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:14.353191 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:14.354181 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:14.354191 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:14.354834 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:14.354837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:14.354995 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:14.355662 (+ 667us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 12:50:14.355663 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:50:14.355779 (+ 116us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 12:50:14.355863 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:50:14.356174 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:50:14.356179 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:50:14.356182 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 12:50:14.356285 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 12:50:14.356535 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":116,"threads_started":1}
I20260927 12:50:14.358952 20453 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41761
I20260927 12:50:14.359170 20453 heartbeater.cc:464] Registering TS with master...
I20260927 12:50:14.359670 20453 heartbeater.cc:511] Master 127.0.98.62:41761 requested a full tablet report, sending...
I20260927 12:50:14.360769 20134 ts_manager.cc:194] Registered new tserver with Master: eb2233ac568342c39c675fbff008197e (127.0.98.2:35515)
I20260927 12:50:14.361326 20134 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:37371
W20260927 12:50:14.463965 20456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:50:14.464250 20456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:50:14.464303 20456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:50:14.468580 20456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:50:14.468652 20456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:50:14.468740 20456 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:50:14.473767 20456 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790513161160093-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790513161160093-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790513161160093-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:50:14.474898 20456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:50:14.476321 20456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:50:14.485483 20462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:50:14.485545 20464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:50:14.485483 20461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:50:14.486234 20456 server_base.cc:1034] running on GCE node
I20260927 12:50:14.486670 20456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:50:14.487346 20456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:50:14.488528 20456 hybrid_clock.cc:648] HybridClock initialized: now 1790513414488512 us; error 29 us; skew 500 ppm
I20260927 12:50:14.490538 20456 webserver.cc:460] Webserver started at http://127.0.98.3:45931/ using document root <none> and password file <none>
I20260927 12:50:14.491109 20456 fs_manager.cc:359] Metadata directory not provided
I20260927 12:50:14.491174 20456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:50:14.491379 20456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:50:14.493430 20456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790513161160093-392-0/minicluster-data/ts-2/data/instance:
uuid: "ba6858b373d546acba0375e00b587268"
format_stamp: "Formatted at 2026-09-27 12:50:14 on dist-test-slave-lkn4"
server_key: "5f6259b93d2b96b757608f6c79ce4554"
server_key_iv: "99c860f00b51d910ca4ba23e949abf28"
server_key_version: "encryptionkey@0"
I20260927 12:50:14.493957 20456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790513161160093-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ba6858b373d546acba0375e00b587268"
format_stamp: "Formatted at 2026-09-27 12:50:14 on dist-test-slave-lkn4"
server_key: "5f6259b93d2b96b757608f6c79ce4554"
server_key_iv: "99c860f00b51d910ca4ba23e949abf28"
server_key_version: "encryptionkey@0"
I20260927 12:50:14.497762 20456 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 12:50:14.500341 20470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:14.501756 20456 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 12:50:14.501880 20456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790513161160093-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790513161160093-392-0/minicluster-data/ts-2/wal
uuid: "ba6858b373d546acba0375e00b587268"
format_stamp: "Formatted at 2026-09-27 12:50:14 on dist-test-slave-lkn4"
server_key: "5f6259b93d2b96b757608f6c79ce4554"
server_key_iv: "99c860f00b51d910ca4ba23e949abf28"
server_key_version: "encryptionkey@0"
I20260927 12:50:14.501984 20456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790513161160093-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790513161160093-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790513161160093-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:50:14.522899 20456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:50:14.523644 20456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:50:14.523847 20456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:50:14.524473 20456 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:50:14.525472 20456 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:50:14.525520 20456 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:14.525574 20456 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:50:14.525602 20456 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:14.534282 20456 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46657
I20260927 12:50:14.534299 20583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46657 every 8 connection(s)
I20260927 12:50:14.535267 20456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790513161160093-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:50:14.536576 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 20456
I20260927 12:50:14.536684 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790513161160093-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:50:14.536978 392 external_mini_cluster.cc:1428] Setting key 754873931701bc9d7d4aa54653e46f7e
I20260927 12:50:14.545071 20173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:14.537212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46593 (local address 127.0.98.62:41761)
0927 12:50:14.537339 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:14.537343 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:14.538689 (+ 1346us) server_negotiation.cc:407] Connection header received
0927 12:50:14.540435 (+ 1746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:14.540437 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:14.540485 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:14.540559 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:14.541465 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:14.541986 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:14.543135 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:14.543358 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:14.543438 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:14.544542 (+ 1104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:14.544545 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:14.544548 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:50:14.544563 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:14.544583 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:50:14.544591 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:14.544672 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:14.544889 (+ 217us) server_negotiation.cc:299] Negotiation successful
0927 12:50:14.544953 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 12:50:14.545933 20586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:14.537727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41761 (local address 127.0.98.3:46593)
0927 12:50:14.538538 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:14.538592 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 12:50:14.540194 (+ 1602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:50:14.540702 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:50:14.540710 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:50:14.540799 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:50:14.541296 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:14.541309 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:14.542111 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:14.542114 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:14.543023 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:14.543032 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:14.543496 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:14.543499 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:14.543647 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:14.544254 (+ 607us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 12:50:14.544256 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:50:14.544299 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 12:50:14.544366 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:50:14.544706 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:50:14.544715 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:50:14.544722 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 12:50:14.544841 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 12:50:14.545085 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":146,"threads_started":1}
I20260927 12:50:14.547184 20584 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41761
I20260927 12:50:14.547436 20584 heartbeater.cc:464] Registering TS with master...
I20260927 12:50:14.548041 20584 heartbeater.cc:511] Master 127.0.98.62:41761 requested a full tablet report, sending...
I20260927 12:50:14.549101 20134 ts_manager.cc:194] Registered new tserver with Master: ba6858b373d546acba0375e00b587268 (127.0.98.3:46657)
I20260927 12:50:14.549656 20134 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:46593
I20260927 12:50:14.551182 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 12:50:14.556629 20173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:14.553094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38450 (local address 127.0.98.62:41761)
0927 12:50:14.553200 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:14.553204 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:14.553366 (+ 162us) server_negotiation.cc:407] Connection header received
0927 12:50:14.553478 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:14.553480 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:14.553519 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:14.553574 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:14.553780 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:14.554328 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:14.555156 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:14.555395 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:14.555473 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:14.556231 (+ 758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:14.556234 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:14.556236 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:50:14.556248 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:14.556266 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:50:14.556273 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:14.556328 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:14.556474 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 12:50:14.556520 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 12:50:14.559654 20134 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38450:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:50:14.562896 20134 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:50:14.577801 20598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:14.572422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35515 (local address 127.0.0.1:48338)
0927 12:50:14.573095 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:14.573108 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:50:14.573181 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:50:14.573707 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:50:14.573710 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:50:14.573728 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:50:14.573826 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:14.573832 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:14.575215 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:14.575219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:14.576344 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:14.576355 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:14.577350 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:14.577353 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:14.577429 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:14.577459 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 12:50:14.577532 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 12:50:14.577645 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"mutex_wait_us":20,"thread_start_us":77,"threads_started":1}
I20260927 12:50:14.577842 20600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:14.572872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48338 (local address 127.0.98.2:35515)
0927 12:50:14.573345 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:14.573360 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:14.573383 (+ 23us) server_negotiation.cc:407] Connection header received
0927 12:50:14.573447 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:14.573450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:14.573588 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:14.573699 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:50:14.573946 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:14.575091 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:14.576469 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:14.577234 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:14.577341 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:14.577515 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:14.577569 (+ 54us) server_negotiation.cc:299] Negotiation successful
0927 12:50:14.577677 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":128,"threads_started":1}
I20260927 12:50:14.577927 20597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:14.572011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46657 (local address 127.0.0.1:50266)
0927 12:50:14.572576 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:14.572588 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:50:14.572684 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:50:14.574393 (+ 1709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:50:14.574396 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:50:14.574414 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:50:14.574497 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:14.574502 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:14.575661 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:14.575665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:14.576635 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:14.576645 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:14.577618 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:14.577622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:14.577693 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:14.577714 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 12:50:14.577775 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 12:50:14.577830 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":132,"threads_started":1}
I20260927 12:50:14.578074 20599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:14.572594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50266 (local address 127.0.98.3:46657)
0927 12:50:14.573945 (+ 1351us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:14.573956 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:14.573977 (+ 21us) server_negotiation.cc:407] Connection header received
0927 12:50:14.574071 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:14.574076 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:14.574261 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:14.574407 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:50:14.574599 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:14.575540 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:14.576775 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:14.577498 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:14.577595 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:14.577757 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:14.577819 (+ 62us) server_negotiation.cc:299] Negotiation successful
0927 12:50:14.577905 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1252,"thread_start_us":181,"threads_started":1}
I20260927 12:50:14.578462 20595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:14.571626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35649 (local address 127.0.0.1:58404)
0927 12:50:14.571990 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:14.572048 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 12:50:14.572174 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:50:14.572960 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:50:14.572966 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:50:14.572989 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:50:14.573119 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:14.573127 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:14.574337 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:14.574340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:14.575403 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:14.575431 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:14.578085 (+ 2654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:14.578088 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:14.578174 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:14.578197 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 12:50:14.578267 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 12:50:14.578337 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":2,"thread_start_us":88,"threads_started":1}
I20260927 12:50:14.578580 20596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:14.571769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58404 (local address 127.0.98.1:35649)
0927 12:50:14.572187 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:14.572197 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:14.572225 (+ 28us) server_negotiation.cc:407] Connection header received
0927 12:50:14.572317 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:14.572323 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:14.572568 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:14.572782 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:50:14.573242 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:14.574237 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:14.575555 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:14.576360 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:14.576460 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:14.576616 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:14.578333 (+ 1717us) server_negotiation.cc:299] Negotiation successful
0927 12:50:14.578434 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":121,"threads_started":1}
I20260927 12:50:14.580698 20387 tablet_service.cc:1467] Processing CreateTablet for tablet b70a7a1ca32d4ebc85555e19aca2435d (DEFAULT_TABLE table=test-table [id=2f595036da18459c833608202aa4160d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:50:14.580791 20518 tablet_service.cc:1467] Processing CreateTablet for tablet b70a7a1ca32d4ebc85555e19aca2435d (DEFAULT_TABLE table=test-table [id=2f595036da18459c833608202aa4160d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:50:14.581717 20387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b70a7a1ca32d4ebc85555e19aca2435d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:50:14.581719 20518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b70a7a1ca32d4ebc85555e19aca2435d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:50:14.583932 20256 tablet_service.cc:1467] Processing CreateTablet for tablet b70a7a1ca32d4ebc85555e19aca2435d (DEFAULT_TABLE table=test-table [id=2f595036da18459c833608202aa4160d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:50:14.584986 20256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b70a7a1ca32d4ebc85555e19aca2435d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:50:14.588538 20602 tablet_bootstrap.cc:492] T b70a7a1ca32d4ebc85555e19aca2435d P eb2233ac568342c39c675fbff008197e: Bootstrap starting.
I20260927 12:50:14.589398 20601 tablet_bootstrap.cc:492] T b70a7a1ca32d4ebc85555e19aca2435d P ba6858b373d546acba0375e00b587268: Bootstrap starting.
I20260927 12:50:14.590957 20602 tablet_bootstrap.cc:654] T b70a7a1ca32d4ebc85555e19aca2435d P eb2233ac568342c39c675fbff008197e: Neither blocks nor log segments found. Creating new log.
I20260927 12:50:14.591391 20603 tablet_bootstrap.cc:492] T b70a7a1ca32d4ebc85555e19aca2435d P 4621ea9b61a846a298d45faeac40009c: Bootstrap starting.
I20260927 12:50:14.591419 20601 tablet_bootstrap.cc:654] T b70a7a1ca32d4ebc85555e19aca2435d P ba6858b373d546acba0375e00b587268: Neither blocks nor log segments found. Creating new log.
I20260927 12:50:14.591831 20602 log.cc:824] T b70a7a1ca32d4ebc85555e19aca2435d P eb2233ac568342c39c675fbff008197e: Log is configured to *not* fsync() on all Append() calls
I20260927 12:50:14.592164 20601 log.cc:824] T b70a7a1ca32d4ebc85555e19aca2435d P ba6858b373d546acba0375e00b587268: Log is configured to *not* fsync() on all Append() calls
I20260927 12:50:14.593470 20603 tablet_bootstrap.cc:654] T b70a7a1ca32d4ebc85555e19aca2435d P 4621ea9b61a846a298d45faeac40009c: Neither blocks nor log segments found. Creating new log.
I20260927 12:50:14.593935 20602 tablet_bootstrap.cc:492] T b70a7a1ca32d4ebc85555e19aca2435d P eb2233ac568342c39c675fbff008197e: No bootstrap required, opened a new log
I20260927 12:50:14.594117 20601 tablet_bootstrap.cc:492] T b70a7a1ca32d4ebc85555e19aca2435d P ba6858b373d546acba0375e00b587268: No bootstrap required, opened a new log
I20260927 12:50:14.594149 20602 ts_tablet_manager.cc:1397] T b70a7a1ca32d4ebc85555e19aca2435d P eb2233ac568342c39c675fbff008197e: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 12:50:14.594298 20601 ts_tablet_manager.cc:1397] T b70a7a1ca32d4ebc85555e19aca2435d P ba6858b373d546acba0375e00b587268: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260927 12:50:14.594281 20603 log.cc:824] T b70a7a1ca32d4ebc85555e19aca2435d P 4621ea9b61a846a298d45faeac40009c: Log is configured to *not* fsync() on all Append() calls
I20260927 12:50:14.596230 20603 tablet_bootstrap.cc:492] T b70a7a1ca32d4ebc85555e19aca2435d P 4621ea9b61a846a298d45faeac40009c: No bootstrap required, opened a new log
I20260927 12:50:14.596467 20603 ts_tablet_manager.cc:1397] T b70a7a1ca32d4ebc85555e19aca2435d P 4621ea9b61a846a298d45faeac40009c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 12:50:14.597759 20602 raft_consensus.cc:348] T b70a7a1ca32d4ebc85555e19aca2435d P eb2233ac568342c39c675fbff008197e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4621ea9b61a846a298d45faeac40009c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35649 } } peers { permanent_uuid: "eb2233ac568342c39c675fbff008197e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35515 } } peers { permanent_uuid: "ba6858b373d546acba0375e00b587268" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46657 } }
I20260927 12:50:14.597971 20602 raft_consensus.cc:374] T b70a7a1ca32d4ebc85555e19aca2435d P eb2233ac568342c39c675fbff008197e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:50:14.598021 20602 raft_consensus.cc:729] T b70a7a1ca32d4ebc85555e19aca2435d P eb2233ac568342c39c675fbff008197e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2233ac568342c39c675fbff008197e, State: Initialized, Role: FOLLOWER
I20260927 12:50:14.598471 20602 consensus_queue.cc:260] T b70a7a1ca32d4ebc85555e19aca2435d P eb2233ac568342c39c675fbff008197e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4621ea9b61a846a298d45faeac40009c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35649 } } peers { permanent_uuid: "eb2233ac568342c39c675fbff008197e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35515 } } peers { permanent_uuid: "ba6858b373d546acba0375e00b587268" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46657 } }
I20260927 12:50:14.598445 20601 raft_consensus.cc:348] T b70a7a1ca32d4ebc85555e19aca2435d P ba6858b373d546acba0375e00b587268 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4621ea9b61a846a298d45faeac40009c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35649 } } peers { permanent_uuid: "eb2233ac568342c39c675fbff008197e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35515 } } peers { permanent_uuid: "ba6858b373d546acba0375e00b587268" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46657 } }
I20260927 12:50:14.598695 20601 raft_consensus.cc:374] T b70a7a1ca32d4ebc85555e19aca2435d P ba6858b373d546acba0375e00b587268 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:50:14.598752 20601 raft_consensus.cc:729] T b70a7a1ca32d4ebc85555e19aca2435d P ba6858b373d546acba0375e00b587268 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba6858b373d546acba0375e00b587268, State: Initialized, Role: FOLLOWER
I20260927 12:50:14.599146 20453 heartbeater.cc:503] Master 127.0.98.62:41761 was elected leader, sending a full tablet report...
I20260927 12:50:14.599213 20601 consensus_queue.cc:260] T b70a7a1ca32d4ebc85555e19aca2435d P ba6858b373d546acba0375e00b587268 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4621ea9b61a846a298d45faeac40009c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35649 } } peers { permanent_uuid: "eb2233ac568342c39c675fbff008197e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35515 } } peers { permanent_uuid: "ba6858b373d546acba0375e00b587268" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46657 } }
I20260927 12:50:14.599493 20602 ts_tablet_manager.cc:1428] T b70a7a1ca32d4ebc85555e19aca2435d P eb2233ac568342c39c675fbff008197e: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20260927 12:50:14.600170 20601 ts_tablet_manager.cc:1428] T b70a7a1ca32d4ebc85555e19aca2435d P ba6858b373d546acba0375e00b587268: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 12:50:14.600201 20584 heartbeater.cc:503] Master 127.0.98.62:41761 was elected leader, sending a full tablet report...
I20260927 12:50:14.600656 20603 raft_consensus.cc:348] T b70a7a1ca32d4ebc85555e19aca2435d P 4621ea9b61a846a298d45faeac40009c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4621ea9b61a846a298d45faeac40009c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35649 } } peers { permanent_uuid: "eb2233ac568342c39c675fbff008197e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35515 } } peers { permanent_uuid: "ba6858b373d546acba0375e00b587268" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46657 } }
I20260927 12:50:14.600903 20603 raft_consensus.cc:374] T b70a7a1ca32d4ebc85555e19aca2435d P 4621ea9b61a846a298d45faeac40009c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:50:14.600951 20603 raft_consensus.cc:729] T b70a7a1ca32d4ebc85555e19aca2435d P 4621ea9b61a846a298d45faeac40009c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4621ea9b61a846a298d45faeac40009c, State: Initialized, Role: FOLLOWER
I20260927 12:50:14.601348 20603 consensus_queue.cc:260] T b70a7a1ca32d4ebc85555e19aca2435d P 4621ea9b61a846a298d45faeac40009c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4621ea9b61a846a298d45faeac40009c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35649 } } peers { permanent_uuid: "eb2233ac568342c39c675fbff008197e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35515 } } peers { permanent_uuid: "ba6858b373d546acba0375e00b587268" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46657 } }
I20260927 12:50:14.602399 20603 ts_tablet_manager.cc:1428] T b70a7a1ca32d4ebc85555e19aca2435d P 4621ea9b61a846a298d45faeac40009c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 12:50:14.602767 20322 heartbeater.cc:503] Master 127.0.98.62:41761 was elected leader, sending a full tablet report...
W20260927 12:50:14.615985 20323 tablet.cc:2369] T b70a7a1ca32d4ebc85555e19aca2435d P 4621ea9b61a846a298d45faeac40009c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 12:50:14.786273 20585 tablet.cc:2369] T b70a7a1ca32d4ebc85555e19aca2435d P ba6858b373d546acba0375e00b587268: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 12:50:14.847478 20454 tablet.cc:2369] T b70a7a1ca32d4ebc85555e19aca2435d P eb2233ac568342c39c675fbff008197e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:50:14.885587 20607 raft_consensus.cc:482] T b70a7a1ca32d4ebc85555e19aca2435d P eb2233ac568342c39c675fbff008197e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:50:14.885802 20607 raft_consensus.cc:504] T b70a7a1ca32d4ebc85555e19aca2435d P eb2233ac568342c39c675fbff008197e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4621ea9b61a846a298d45faeac40009c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35649 } } peers { permanent_uuid: "eb2233ac568342c39c675fbff008197e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35515 } } peers { permanent_uuid: "ba6858b373d546acba0375e00b587268" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46657 } }
I20260927 12:50:14.887053 20607 leader_election.cc:288] T b70a7a1ca32d4ebc85555e19aca2435d P eb2233ac568342c39c675fbff008197e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4621ea9b61a846a298d45faeac40009c (127.0.98.1:35649), ba6858b373d546acba0375e00b587268 (127.0.98.3:46657)
I20260927 12:50:14.891067 20596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:14.887381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33677 (local address 127.0.98.1:35649)
0927 12:50:14.887506 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:14.887511 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:14.887738 (+ 227us) server_negotiation.cc:407] Connection header received
0927 12:50:14.887911 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:14.887915 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:14.887965 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:14.888028 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:50:14.888353 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:14.888985 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:14.890100 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:14.890499 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:14.890593 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:14.890648 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:14.890865 (+ 217us) server_negotiation.cc:299] Negotiation successful
0927 12:50:14.890939 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 12:50:14.891067 20599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:14.887464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43207 (local address 127.0.98.3:46657)
0927 12:50:14.887615 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:14.887619 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:14.887858 (+ 239us) server_negotiation.cc:407] Connection header received
0927 12:50:14.888030 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:14.888033 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:14.888084 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:14.888199 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:50:14.888427 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:14.889011 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:14.890085 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:14.890485 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:14.890587 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:14.890647 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:14.890868 (+ 221us) server_negotiation.cc:299] Negotiation successful
0927 12:50:14.890937 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 12:50:14.891098 20610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:14.887296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35649 (local address 127.0.98.2:33677)
0927 12:50:14.887641 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:14.887665 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 12:50:14.887775 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:50:14.888088 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:50:14.888091 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:50:14.888107 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:50:14.888236 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:14.888243 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:14.889080 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:14.889084 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:14.889956 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:14.889965 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:14.890611 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:14.890614 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:14.890753 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:14.890771 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 12:50:14.890833 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 12:50:14.890937 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":10,"thread_start_us":113,"threads_started":1}
I20260927 12:50:14.891119 20611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:14.887352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46657 (local address 127.0.98.2:43207)
0927 12:50:14.887774 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:14.887799 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 12:50:14.887900 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:50:14.888210 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:50:14.888212 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:50:14.888227 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:50:14.888329 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:14.888335 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:14.889115 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:14.889117 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:14.889932 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:14.889943 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:14.890602 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:14.890606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:14.890757 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:14.890775 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 12:50:14.890840 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 12:50:14.890987 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"mutex_wait_us":82,"thread_start_us":89,"threads_started":1}
I20260927 12:50:14.891691 20276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b70a7a1ca32d4ebc85555e19aca2435d" candidate_uuid: "eb2233ac568342c39c675fbff008197e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4621ea9b61a846a298d45faeac40009c" is_pre_election: true
I20260927 12:50:14.892006 20276 raft_consensus.cc:2454] T b70a7a1ca32d4ebc85555e19aca2435d P 4621ea9b61a846a298d45faeac40009c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2233ac568342c39c675fbff008197e in term 0.
I20260927 12:50:14.892032 20538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b70a7a1ca32d4ebc85555e19aca2435d" candidate_uuid: "eb2233ac568342c39c675fbff008197e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba6858b373d546acba0375e00b587268" is_pre_election: true
I20260927 12:50:14.892305 20538 raft_consensus.cc:2454] T b70a7a1ca32d4ebc85555e19aca2435d P ba6858b373d546acba0375e00b587268 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2233ac568342c39c675fbff008197e in term 0.
I20260927 12:50:14.892499 20341 leader_election.cc:302] T b70a7a1ca32d4ebc85555e19aca2435d P eb2233ac568342c39c675fbff008197e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4621ea9b61a846a298d45faeac40009c, eb2233ac568342c39c675fbff008197e; no voters:
I20260927 12:50:14.892860 20607 raft_consensus.cc:2781] T b70a7a1ca32d4ebc85555e19aca2435d P eb2233ac568342c39c675fbff008197e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:50:14.892947 20607 raft_consensus.cc:482] T b70a7a1ca32d4ebc85555e19aca2435d P eb2233ac568342c39c675fbff008197e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:50:14.892994 20607 raft_consensus.cc:3037] T b70a7a1ca32d4ebc85555e19aca2435d P eb2233ac568342c39c675fbff008197e [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:50:14.893975 20607 raft_consensus.cc:504] T b70a7a1ca32d4ebc85555e19aca2435d P eb2233ac568342c39c675fbff008197e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4621ea9b61a846a298d45faeac40009c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35649 } } peers { permanent_uuid: "eb2233ac568342c39c675fbff008197e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35515 } } peers { permanent_uuid: "ba6858b373d546acba0375e00b587268" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46657 } }
I20260927 12:50:14.894349 20607 leader_election.cc:288] T b70a7a1ca32d4ebc85555e19aca2435d P eb2233ac568342c39c675fbff008197e [CANDIDATE]: Term 1 election: Requested vote from peers 4621ea9b61a846a298d45faeac40009c (127.0.98.1:35649), ba6858b373d546acba0375e00b587268 (127.0.98.3:46657)
I20260927 12:50:14.894691 20276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b70a7a1ca32d4ebc85555e19aca2435d" candidate_uuid: "eb2233ac568342c39c675fbff008197e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4621ea9b61a846a298d45faeac40009c"
I20260927 12:50:14.894695 20538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b70a7a1ca32d4ebc85555e19aca2435d" candidate_uuid: "eb2233ac568342c39c675fbff008197e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba6858b373d546acba0375e00b587268"
I20260927 12:50:14.894806 20276 raft_consensus.cc:3037] T b70a7a1ca32d4ebc85555e19aca2435d P 4621ea9b61a846a298d45faeac40009c [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:50:14.894817 20538 raft_consensus.cc:3037] T b70a7a1ca32d4ebc85555e19aca2435d P ba6858b373d546acba0375e00b587268 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:50:14.895823 20276 raft_consensus.cc:2454] T b70a7a1ca32d4ebc85555e19aca2435d P 4621ea9b61a846a298d45faeac40009c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2233ac568342c39c675fbff008197e in term 1.
I20260927 12:50:14.895915 20538 raft_consensus.cc:2454] T b70a7a1ca32d4ebc85555e19aca2435d P ba6858b373d546acba0375e00b587268 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb2233ac568342c39c675fbff008197e in term 1.
I20260927 12:50:14.896128 20341 leader_election.cc:302] T b70a7a1ca32d4ebc85555e19aca2435d P eb2233ac568342c39c675fbff008197e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4621ea9b61a846a298d45faeac40009c, eb2233ac568342c39c675fbff008197e; no voters:
I20260927 12:50:14.896299 20607 raft_consensus.cc:2781] T b70a7a1ca32d4ebc85555e19aca2435d P eb2233ac568342c39c675fbff008197e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:50:14.896510 20607 raft_consensus.cc:686] T b70a7a1ca32d4ebc85555e19aca2435d P eb2233ac568342c39c675fbff008197e [term 1 LEADER]: Becoming Leader. State: Replica: eb2233ac568342c39c675fbff008197e, State: Running, Role: LEADER
I20260927 12:50:14.896823 20607 consensus_queue.cc:237] T b70a7a1ca32d4ebc85555e19aca2435d P eb2233ac568342c39c675fbff008197e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4621ea9b61a846a298d45faeac40009c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35649 } } peers { permanent_uuid: "eb2233ac568342c39c675fbff008197e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35515 } } peers { permanent_uuid: "ba6858b373d546acba0375e00b587268" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46657 } }
I20260927 12:50:14.899964 20132 catalog_manager.cc:5495] T b70a7a1ca32d4ebc85555e19aca2435d P eb2233ac568342c39c675fbff008197e reported cstate change: term changed from 0 to 1, leader changed from <none> to eb2233ac568342c39c675fbff008197e (127.0.98.2). New cstate: current_term: 1 leader_uuid: "eb2233ac568342c39c675fbff008197e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4621ea9b61a846a298d45faeac40009c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb2233ac568342c39c675fbff008197e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ba6858b373d546acba0375e00b587268" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46657 } health_report { overall_health: UNKNOWN } } }
I20260927 12:50:14.943701 20600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:14.940096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48350 (local address 127.0.98.2:35515)
0927 12:50:14.940239 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:14.940244 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:14.940258 (+ 14us) server_negotiation.cc:407] Connection header received
0927 12:50:14.940364 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:14.940367 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:14.940425 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:14.940518 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:50:14.940781 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:14.941287 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:14.942252 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:14.942467 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:14.942553 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:14.942863 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:50:14.943304 (+ 441us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:50:14.943414 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:14.943491 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 12:50:14.943549 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 12:50:14.950453 20276 raft_consensus.cc:1261] T b70a7a1ca32d4ebc85555e19aca2435d P 4621ea9b61a846a298d45faeac40009c [term 1 FOLLOWER]: Refusing update from remote peer eb2233ac568342c39c675fbff008197e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:50:14.950453 20538 raft_consensus.cc:1261] T b70a7a1ca32d4ebc85555e19aca2435d P ba6858b373d546acba0375e00b587268 [term 1 FOLLOWER]: Refusing update from remote peer eb2233ac568342c39c675fbff008197e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:50:14.951341 20612 consensus_queue.cc:1035] T b70a7a1ca32d4ebc85555e19aca2435d P eb2233ac568342c39c675fbff008197e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba6858b373d546acba0375e00b587268" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:50:14.951545 20607 consensus_queue.cc:1035] T b70a7a1ca32d4ebc85555e19aca2435d P eb2233ac568342c39c675fbff008197e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4621ea9b61a846a298d45faeac40009c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:50:14.959100 20615 mvcc.cc:205] Tried to move back new op lower bound from 7333942947628040192 to 7333942947420086272. Current Snapshot: MvccSnapshot[applied={T|T < 7333942947628040192}]
I20260927 12:50:14.959687 20617 mvcc.cc:205] Tried to move back new op lower bound from 7333942947628040192 to 7333942947420086272. Current Snapshot: MvccSnapshot[applied={T|T < 7333942947628040192}]
I20260927 12:50:14.968397 20134 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:50:14.968604 20134 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38450:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:50:14.971252 20134 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c92c03b0e4bc403097eecbf37fb1f42e: Sending DeleteTablet for 3 replicas of tablet b70a7a1ca32d4ebc85555e19aca2435d
I20260927 12:50:14.972229 20387 tablet_service.cc:1514] Processing DeleteTablet for tablet b70a7a1ca32d4ebc85555e19aca2435d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:50:14 UTC) from {username='slave'} at 127.0.0.1:48338
I20260927 12:50:14.972236 20518 tablet_service.cc:1514] Processing DeleteTablet for tablet b70a7a1ca32d4ebc85555e19aca2435d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:50:14 UTC) from {username='slave'} at 127.0.0.1:50266
I20260927 12:50:14.972268 20256 tablet_service.cc:1514] Processing DeleteTablet for tablet b70a7a1ca32d4ebc85555e19aca2435d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:50:14 UTC) from {username='slave'} at 127.0.0.1:58404
I20260927 12:50:14.973007 20625 tablet_replica.cc:330] T b70a7a1ca32d4ebc85555e19aca2435d P ba6858b373d546acba0375e00b587268: stopping tablet replica
I20260927 12:50:14.973363 20625 raft_consensus.cc:2229] T b70a7a1ca32d4ebc85555e19aca2435d P ba6858b373d546acba0375e00b587268 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:50:14.973490 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 20194
I20260927 12:50:14.973726 20625 raft_consensus.cc:2258] T b70a7a1ca32d4ebc85555e19aca2435d P ba6858b373d546acba0375e00b587268 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:50:14.974155 20626 tablet_replica.cc:330] T b70a7a1ca32d4ebc85555e19aca2435d P eb2233ac568342c39c675fbff008197e: stopping tablet replica
I20260927 12:50:14.974462 20626 raft_consensus.cc:2229] T b70a7a1ca32d4ebc85555e19aca2435d P eb2233ac568342c39c675fbff008197e [term 1 LEADER]: Raft consensus shutting down.
I20260927 12:50:14.974869 20626 raft_consensus.cc:2258] T b70a7a1ca32d4ebc85555e19aca2435d P eb2233ac568342c39c675fbff008197e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:50:14.975788 20625 ts_tablet_manager.cc:1905] T b70a7a1ca32d4ebc85555e19aca2435d P ba6858b373d546acba0375e00b587268: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:50:14.978014 20625 ts_tablet_manager.cc:1918] T b70a7a1ca32d4ebc85555e19aca2435d P ba6858b373d546acba0375e00b587268: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:50:14.978127 20625 log.cc:1196] T b70a7a1ca32d4ebc85555e19aca2435d P ba6858b373d546acba0375e00b587268: Deleting WAL directory at /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790513161160093-392-0/minicluster-data/ts-2/wal/wals/b70a7a1ca32d4ebc85555e19aca2435d
I20260927 12:50:14.978340 20625 ts_tablet_manager.cc:1939] T b70a7a1ca32d4ebc85555e19aca2435d P ba6858b373d546acba0375e00b587268: Deleting consensus metadata
I20260927 12:50:14.979429 20118 catalog_manager.cc:4841] TS ba6858b373d546acba0375e00b587268 (127.0.98.3:46657): tablet b70a7a1ca32d4ebc85555e19aca2435d (table test-table [id=2f595036da18459c833608202aa4160d]) successfully deleted
I20260927 12:50:14.980185 20626 ts_tablet_manager.cc:1905] T b70a7a1ca32d4ebc85555e19aca2435d P eb2233ac568342c39c675fbff008197e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 12:50:14.982115 20119 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35649 (error 108)
I20260927 12:50:14.982314 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 20325
I20260927 12:50:14.984279 20598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:14.983946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35649 (local address 127.0.0.1:58406)
0927 12:50:14.984091 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:14.984187 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 12:50:14.984465 20119 catalog_manager.cc:4584] TS 4621ea9b61a846a298d45faeac40009c (127.0.98.1:35649): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b70a7a1ca32d4ebc85555e19aca2435d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35649: connect: Connection refused (error 111)
I20260927 12:50:14.987493 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 20456
I20260927 12:50:14.988279 20598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:14.987671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35515 (local address 127.0.0.1:48366)
0927 12:50:14.988124 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:14.988188 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:35515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374}
I20260927 12:50:14.992267 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 20103
2026-09-27T12:50:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1322 ms)
[----------] 6 tests from EncryptionPolicyTest (10219 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790513161160093-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:50:15 dist-test-slave-lkn4 krb5kdc[20630](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:50:15 dist-test-slave-lkn4 krb5kdc[20630](info): set up 2 sockets
Sep 27 12:50:15 dist-test-slave-lkn4 krb5kdc[20630](info): commencing operation
krb5kdc: starting...
W20260927 12:50:15.970221 392 mini_kdc.cc:121] Time spent starting KDC: real 0.957s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790513161160093-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790513161160093-392-0/krb5kdc/test-user.keytab.
Sep 27 12:50:16 dist-test-slave-lkn4 krb5kdc[20630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:50:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:50:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790513161160093-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790513161160093-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 12:50:16.091041 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790513161160093-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790513161160093-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790513161160093-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37315
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790513161160093-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37315
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:50:16.198764 20643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:50:16.199043 20643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:50:16.199101 20643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:50:16.203092 20643 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:50:16.203154 20643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:50:16.203179 20643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:50:16.203213 20643 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:50:16.203248 20643 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:50:16.208636 20643 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790513161160093-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790513161160093-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37315
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790513161160093-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790513161160093-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:50:16.209681 20643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:50:16.210701 20643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:50:16.219177 20649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:50:16.219321 20643 server_base.cc:1034] running on GCE node
W20260927 12:50:16.219201 20651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:50:16.219453 20648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:50:16.220049 20643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:50:16.221134 20643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:50:16.222286 20643 hybrid_clock.cc:648] HybridClock initialized: now 1790513416222266 us; error 31 us; skew 500 ppm
Sep 27 12:50:16 dist-test-slave-lkn4 krb5kdc[20630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513416, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:50:16.224931 20643 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 12:50:16.225823 20643 webserver.cc:460] Webserver started at http://127.0.0.1:33331/ using document root <none> and password file <none>
I20260927 12:50:16.226387 20643 fs_manager.cc:359] Metadata directory not provided
I20260927 12:50:16.226454 20643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:50:16.226667 20643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:50:16.228662 20643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790513161160093-392-0/minicluster-data/master-0/data/instance:
uuid: "d4ef2f9f69a34343b066eba6dbecf3c7"
format_stamp: "Formatted at 2026-09-27 12:50:16 on dist-test-slave-lkn4"
server_key: "1a0d23b858414c39b2054627b791bea0"
server_key_iv: "e3ec611e7b52d7ad0333272610254c7d"
server_key_version: "encryptionkey@0"
I20260927 12:50:16.229182 20643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790513161160093-392-0/minicluster-data/master-0/wal/instance:
uuid: "d4ef2f9f69a34343b066eba6dbecf3c7"
format_stamp: "Formatted at 2026-09-27 12:50:16 on dist-test-slave-lkn4"
server_key: "1a0d23b858414c39b2054627b791bea0"
server_key_iv: "e3ec611e7b52d7ad0333272610254c7d"
server_key_version: "encryptionkey@0"
I20260927 12:50:16.232928 20643 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 12:50:16.235447 20658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:16.236805 20643 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 12:50:16.236928 20643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790513161160093-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790513161160093-392-0/minicluster-data/master-0/wal
uuid: "d4ef2f9f69a34343b066eba6dbecf3c7"
format_stamp: "Formatted at 2026-09-27 12:50:16 on dist-test-slave-lkn4"
server_key: "1a0d23b858414c39b2054627b791bea0"
server_key_iv: "e3ec611e7b52d7ad0333272610254c7d"
server_key_version: "encryptionkey@0"
I20260927 12:50:16.237030 20643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790513161160093-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790513161160093-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790513161160093-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:50:16.253562 20643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:50:16.256511 20643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:50:16.256690 20643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:50:16.263360 20643 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37315
I20260927 12:50:16.263373 20710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37315 every 8 connection(s)
I20260927 12:50:16.264268 20643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:50:16.266933 20711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:50:16.267388 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 20643
I20260927 12:50:16.267515 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790513161160093-392-0/minicluster-data/master-0/wal/instance
I20260927 12:50:16.267807 392 external_mini_cluster.cc:1428] Setting key 30270992726b6613982f6c0d9dbb948a
I20260927 12:50:16.272785 20711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4ef2f9f69a34343b066eba6dbecf3c7: Bootstrap starting.
I20260927 12:50:16.274991 20711 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4ef2f9f69a34343b066eba6dbecf3c7: Neither blocks nor log segments found. Creating new log.
I20260927 12:50:16.275712 20711 log.cc:824] T 00000000000000000000000000000000 P d4ef2f9f69a34343b066eba6dbecf3c7: Log is configured to *not* fsync() on all Append() calls
Sep 27 12:50:16 dist-test-slave-lkn4 krb5kdc[20630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 12:50:16.277717 20711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4ef2f9f69a34343b066eba6dbecf3c7: No bootstrap required, opened a new log
I20260927 12:50:16.281142 20711 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4ef2f9f69a34343b066eba6dbecf3c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4ef2f9f69a34343b066eba6dbecf3c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37315 } }
I20260927 12:50:16.281491 20711 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4ef2f9f69a34343b066eba6dbecf3c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:50:16.281577 20711 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4ef2f9f69a34343b066eba6dbecf3c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4ef2f9f69a34343b066eba6dbecf3c7, State: Initialized, Role: FOLLOWER
I20260927 12:50:16.281682 20714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:16.269241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39355 (local address 127.0.0.1:37315)
0927 12:50:16.269686 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:16.269702 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:16.269740 (+ 38us) server_negotiation.cc:407] Connection header received
0927 12:50:16.270449 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:16.270468 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:16.270808 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:16.271111 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:16.271916 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:16.272960 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:16.273788 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:16.274305 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:16.274449 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:16.277077 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:16.277081 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:16.277095 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:50:16.277120 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:16.278763 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:16.279224 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:16.279229 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:16.279233 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:16.279340 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:16.279659 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:16.279662 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:16.279664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:16.280036 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:16.280240 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:16.280547 (+ 307us) server_negotiation.cc:299] Negotiation successful
0927 12:50:16.280828 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":112,"threads_started":1}
I20260927 12:50:16.282232 20711 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4ef2f9f69a34343b066eba6dbecf3c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4ef2f9f69a34343b066eba6dbecf3c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37315 } }
I20260927 12:50:16.282397 20711 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4ef2f9f69a34343b066eba6dbecf3c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:50:16.282456 20711 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4ef2f9f69a34343b066eba6dbecf3c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:50:16.282543 20711 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4ef2f9f69a34343b066eba6dbecf3c7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:50:16.283504 20711 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4ef2f9f69a34343b066eba6dbecf3c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4ef2f9f69a34343b066eba6dbecf3c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37315 } }
I20260927 12:50:16.283855 20711 leader_election.cc:302] T 00000000000000000000000000000000 P d4ef2f9f69a34343b066eba6dbecf3c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4ef2f9f69a34343b066eba6dbecf3c7; no voters:
I20260927 12:50:16.284190 20711 leader_election.cc:288] T 00000000000000000000000000000000 P d4ef2f9f69a34343b066eba6dbecf3c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:50:16.284276 20716 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4ef2f9f69a34343b066eba6dbecf3c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:50:16.284457 20716 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4ef2f9f69a34343b066eba6dbecf3c7 [term 1 LEADER]: Becoming Leader. State: Replica: d4ef2f9f69a34343b066eba6dbecf3c7, State: Running, Role: LEADER
I20260927 12:50:16.284778 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:50:16.284747 20716 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4ef2f9f69a34343b066eba6dbecf3c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4ef2f9f69a34343b066eba6dbecf3c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37315 } }
I20260927 12:50:16.285377 20711 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4ef2f9f69a34343b066eba6dbecf3c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:50:16.286361 20718 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4ef2f9f69a34343b066eba6dbecf3c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4ef2f9f69a34343b066eba6dbecf3c7. Latest consensus state: current_term: 1 leader_uuid: "d4ef2f9f69a34343b066eba6dbecf3c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4ef2f9f69a34343b066eba6dbecf3c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37315 } } }
I20260927 12:50:16.286484 20718 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4ef2f9f69a34343b066eba6dbecf3c7 [sys.catalog]: This master's current role is: LEADER
I20260927 12:50:16.286742 20717 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4ef2f9f69a34343b066eba6dbecf3c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4ef2f9f69a34343b066eba6dbecf3c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4ef2f9f69a34343b066eba6dbecf3c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37315 } } }
I20260927 12:50:16.286839 20717 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4ef2f9f69a34343b066eba6dbecf3c7 [sys.catalog]: This master's current role is: LEADER
I20260927 12:50:16.286823 20729 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:50:16.290019 20729 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:50:16.295936 20729 catalog_manager.cc:1347] Generated new cluster ID: 108b1bc63c184422aba4e46fcb3d65f4
I20260927 12:50:16.296054 20729 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:50:16.296444 20714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:16.287473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53225 (local address 127.0.0.1:37315)
0927 12:50:16.287573 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:16.287577 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:16.287756 (+ 179us) server_negotiation.cc:407] Connection header received
0927 12:50:16.287895 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:16.287898 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:16.287963 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:16.288055 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:16.288825 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:16.289557 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:16.290676 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:16.290988 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:16.291087 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:16.292798 (+ 1711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:16.292802 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:16.292804 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:50:16.292828 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:16.294595 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:16.295034 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:16.295036 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:16.295037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:16.295107 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:16.295420 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:16.295422 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:16.295423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:16.295581 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:16.295704 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:16.296214 (+ 510us) server_negotiation.cc:299] Negotiation successful
0927 12:50:16.296302 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 12:50:16.316058 20729 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:50:16.317099 20729 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:50:16.327105 20729 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4ef2f9f69a34343b066eba6dbecf3c7: Generated new TSK 0
I20260927 12:50:16.327920 20729 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:50:16.334122 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 20643
W20260927 12:50:16.338853 20628 connection.cc:535] client connection to 127.0.0.1:37315 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37315): Connection reset by peer (error 104)
2026-09-27T12:50:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1340 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790513161160093-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:50:16 dist-test-slave-lkn4 krb5kdc[20742](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:50:16 dist-test-slave-lkn4 krb5kdc[20742](info): set up 2 sockets
Sep 27 12:50:16 dist-test-slave-lkn4 krb5kdc[20742](info): commencing operation
krb5kdc: starting...
W20260927 12:50:17.294131 392 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790513161160093-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790513161160093-392-0/krb5kdc/test-user.keytab.
Sep 27 12:50:17 dist-test-slave-lkn4 krb5kdc[20742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513417, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:50:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:50:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790513161160093-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790513161160093-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 12:50:17.417376 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790513161160093-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790513161160093-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790513161160093-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34221
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790513161160093-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34221
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:50:17.528748 20755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:50:17.529034 20755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:50:17.529075 20755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:50:17.533193 20755 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:50:17.533262 20755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:50:17.533280 20755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:50:17.533298 20755 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:50:17.533310 20755 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:50:17.538707 20755 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790513161160093-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790513161160093-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34221
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790513161160093-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790513161160093-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:50:17.539834 20755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:50:17.540932 20755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:50:17.549475 20760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:50:17.549469 20761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:50:17.549467 20763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:50:17.549809 20755 server_base.cc:1034] running on GCE node
I20260927 12:50:17.550427 20755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:50:17.551527 20755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:50:17.552682 20755 hybrid_clock.cc:648] HybridClock initialized: now 1790513417552668 us; error 31 us; skew 500 ppm
Sep 27 12:50:17 dist-test-slave-lkn4 krb5kdc[20742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513417, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:50:17.555466 20755 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 12:50:17.556331 20755 webserver.cc:460] Webserver started at http://127.0.0.1:45853/ using document root <none> and password file <none>
I20260927 12:50:17.556866 20755 fs_manager.cc:359] Metadata directory not provided
I20260927 12:50:17.556928 20755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:50:17.557111 20755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:50:17.559152 20755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790513161160093-392-0/minicluster-data/master-0/data/instance:
uuid: "3f284b1c56804777a430bdad1fdfb79d"
format_stamp: "Formatted at 2026-09-27 12:50:17 on dist-test-slave-lkn4"
server_key: "fb81a5c8f2739deacd7d0b3e9cfeec38"
server_key_iv: "e6929a0c228b809d58c52c83ecebed2b"
server_key_version: "encryptionkey@0"
I20260927 12:50:17.559691 20755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790513161160093-392-0/minicluster-data/master-0/wal/instance:
uuid: "3f284b1c56804777a430bdad1fdfb79d"
format_stamp: "Formatted at 2026-09-27 12:50:17 on dist-test-slave-lkn4"
server_key: "fb81a5c8f2739deacd7d0b3e9cfeec38"
server_key_iv: "e6929a0c228b809d58c52c83ecebed2b"
server_key_version: "encryptionkey@0"
I20260927 12:50:17.563378 20755 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 12:50:17.565929 20770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:17.567695 20755 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:50:17.567838 20755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790513161160093-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790513161160093-392-0/minicluster-data/master-0/wal
uuid: "3f284b1c56804777a430bdad1fdfb79d"
format_stamp: "Formatted at 2026-09-27 12:50:17 on dist-test-slave-lkn4"
server_key: "fb81a5c8f2739deacd7d0b3e9cfeec38"
server_key_iv: "e6929a0c228b809d58c52c83ecebed2b"
server_key_version: "encryptionkey@0"
I20260927 12:50:17.567955 20755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790513161160093-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790513161160093-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790513161160093-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:50:17.582674 20755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:50:17.586009 20755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:50:17.586248 20755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:50:17.593515 20755 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34221
I20260927 12:50:17.593526 20822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34221 every 8 connection(s)
I20260927 12:50:17.594556 20755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:50:17.597353 20823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:50:17.602886 20823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f284b1c56804777a430bdad1fdfb79d: Bootstrap starting.
I20260927 12:50:17.604043 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 20755
I20260927 12:50:17.604187 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790513161160093-392-0/minicluster-data/master-0/wal/instance
I20260927 12:50:17.604442 392 external_mini_cluster.cc:1428] Setting key d1ab8fe2d859b7c0e7572114b6d4c612
I20260927 12:50:17.605602 20823 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f284b1c56804777a430bdad1fdfb79d: Neither blocks nor log segments found. Creating new log.
I20260927 12:50:17.606390 20823 log.cc:824] T 00000000000000000000000000000000 P 3f284b1c56804777a430bdad1fdfb79d: Log is configured to *not* fsync() on all Append() calls
I20260927 12:50:17.608366 20823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f284b1c56804777a430bdad1fdfb79d: No bootstrap required, opened a new log
I20260927 12:50:17.612500 20823 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f284b1c56804777a430bdad1fdfb79d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f284b1c56804777a430bdad1fdfb79d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34221 } }
I20260927 12:50:17.612804 20823 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f284b1c56804777a430bdad1fdfb79d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:50:17.612888 20823 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f284b1c56804777a430bdad1fdfb79d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f284b1c56804777a430bdad1fdfb79d, State: Initialized, Role: FOLLOWER
I20260927 12:50:17.613374 20823 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f284b1c56804777a430bdad1fdfb79d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f284b1c56804777a430bdad1fdfb79d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34221 } }
I20260927 12:50:17.613503 20823 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f284b1c56804777a430bdad1fdfb79d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:50:17.613565 20823 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f284b1c56804777a430bdad1fdfb79d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:50:17.613655 20826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:17.605578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50299 (local address 127.0.0.1:34221)
0927 12:50:17.606012 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:17.606028 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:17.606113 (+ 85us) server_negotiation.cc:407] Connection header received
0927 12:50:17.606847 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:17.606868 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:17.607215 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:17.607498 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:17.607817 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:17.608800 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:17.610129 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:17.610626 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:17.610829 (+ 203us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:17.611782 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:17.611788 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:17.611802 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:50:17.612210 (+ 408us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:17.612266 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:50:17.612289 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:17.612402 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:17.612579 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 12:50:17.612787 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":121,"threads_started":1}
I20260927 12:50:17.613660 20823 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f284b1c56804777a430bdad1fdfb79d [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:50:17.614816 20823 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f284b1c56804777a430bdad1fdfb79d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f284b1c56804777a430bdad1fdfb79d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34221 } }
I20260927 12:50:17.615160 20823 leader_election.cc:302] T 00000000000000000000000000000000 P 3f284b1c56804777a430bdad1fdfb79d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f284b1c56804777a430bdad1fdfb79d; no voters:
I20260927 12:50:17.615455 20823 leader_election.cc:288] T 00000000000000000000000000000000 P 3f284b1c56804777a430bdad1fdfb79d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:50:17.615557 20828 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f284b1c56804777a430bdad1fdfb79d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:50:17.615837 20828 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f284b1c56804777a430bdad1fdfb79d [term 1 LEADER]: Becoming Leader. State: Replica: 3f284b1c56804777a430bdad1fdfb79d, State: Running, Role: LEADER
I20260927 12:50:17.616205 20828 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f284b1c56804777a430bdad1fdfb79d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f284b1c56804777a430bdad1fdfb79d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34221 } }
I20260927 12:50:17.616700 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:50:17.616832 20823 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f284b1c56804777a430bdad1fdfb79d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:50:17.617812 20829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f284b1c56804777a430bdad1fdfb79d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f284b1c56804777a430bdad1fdfb79d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f284b1c56804777a430bdad1fdfb79d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34221 } } }
I20260927 12:50:17.617947 20829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f284b1c56804777a430bdad1fdfb79d [sys.catalog]: This master's current role is: LEADER
I20260927 12:50:17.618446 20840 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:50:17.618723 20830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f284b1c56804777a430bdad1fdfb79d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f284b1c56804777a430bdad1fdfb79d. Latest consensus state: current_term: 1 leader_uuid: "3f284b1c56804777a430bdad1fdfb79d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f284b1c56804777a430bdad1fdfb79d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34221 } } }
I20260927 12:50:17.618867 20830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f284b1c56804777a430bdad1fdfb79d [sys.catalog]: This master's current role is: LEADER
I20260927 12:50:17.621894 20840 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:50:17.624037 20826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:17.619421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44535 (local address 127.0.0.1:34221)
0927 12:50:17.619520 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:17.619524 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:17.619800 (+ 276us) server_negotiation.cc:407] Connection header received
0927 12:50:17.619924 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:17.619927 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:17.619972 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:17.620049 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:17.620337 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:17.621072 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:17.622245 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:17.622572 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:17.622666 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:17.623585 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:17.623588 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:17.623591 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:50:17.623604 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:17.623631 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:50:17.623644 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:17.623730 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:17.623877 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 12:50:17.623912 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260927 12:50:17.628371 20840 catalog_manager.cc:1347] Generated new cluster ID: 31e22a2a2f3740d1be9f90004e72dfe4
I20260927 12:50:17.628445 20840 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:50:17.640097 20840 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:50:17.641484 20840 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:50:17.648981 20840 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f284b1c56804777a430bdad1fdfb79d: Generated new TSK 0
I20260927 12:50:17.649850 20840 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:50:17.660282 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 20755
W20260927 12:50:17.665082 20740 connection.cc:535] client connection to 127.0.0.1:34221 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34221): Connection reset by peer (error 104)
2026-09-27T12:50:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1326 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790513161160093-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:50:17 dist-test-slave-lkn4 krb5kdc[20854](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:50:17 dist-test-slave-lkn4 krb5kdc[20854](info): set up 2 sockets
Sep 27 12:50:17 dist-test-slave-lkn4 krb5kdc[20854](info): commencing operation
krb5kdc: starting...
W20260927 12:50:18.625959 392 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790513161160093-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790513161160093-392-0/krb5kdc/test-user.keytab.
Sep 27 12:50:18 dist-test-slave-lkn4 krb5kdc[20854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:50:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:50:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790513161160093-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790513161160093-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 12:50:18.749881 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790513161160093-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790513161160093-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790513161160093-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33037
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790513161160093-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33037
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:50:18.859994 20867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:50:18.860275 20867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:50:18.860317 20867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:50:18.864316 20867 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:50:18.864382 20867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:50:18.864401 20867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:50:18.864415 20867 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:50:18.864429 20867 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:50:18.869720 20867 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790513161160093-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790513161160093-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33037
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790513161160093-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790513161160093-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:50:18.870779 20867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:50:18.871762 20867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 12:50:18.880278 20867 server_base.cc:1034] running on GCE node
W20260927 12:50:18.880398 20872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:50:18.880261 20873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:50:18.880626 20875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:50:18.881199 20867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:50:18.882390 20867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:50:18.883558 20867 hybrid_clock.cc:648] HybridClock initialized: now 1790513418883533 us; error 50 us; skew 500 ppm
Sep 27 12:50:18 dist-test-slave-lkn4 krb5kdc[20854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513418, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:50:18.886381 20867 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 12:50:18.887305 20867 webserver.cc:460] Webserver started at http://127.0.0.1:36409/ using document root <none> and password file <none>
I20260927 12:50:18.887881 20867 fs_manager.cc:359] Metadata directory not provided
I20260927 12:50:18.887944 20867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:50:18.888139 20867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:50:18.890345 20867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790513161160093-392-0/minicluster-data/master-0/data/instance:
uuid: "ae4eab09c731424ab0961352468cde79"
format_stamp: "Formatted at 2026-09-27 12:50:18 on dist-test-slave-lkn4"
server_key: "aa943d13ec1fd7bd007130489b688693"
server_key_iv: "3b30946ae49ceaec2ffb7fb82d9230e8"
server_key_version: "encryptionkey@0"
I20260927 12:50:18.890920 20867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790513161160093-392-0/minicluster-data/master-0/wal/instance:
uuid: "ae4eab09c731424ab0961352468cde79"
format_stamp: "Formatted at 2026-09-27 12:50:18 on dist-test-slave-lkn4"
server_key: "aa943d13ec1fd7bd007130489b688693"
server_key_iv: "3b30946ae49ceaec2ffb7fb82d9230e8"
server_key_version: "encryptionkey@0"
I20260927 12:50:18.894830 20867 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 12:50:18.897511 20882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:18.899010 20867 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 12:50:18.899137 20867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790513161160093-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790513161160093-392-0/minicluster-data/master-0/wal
uuid: "ae4eab09c731424ab0961352468cde79"
format_stamp: "Formatted at 2026-09-27 12:50:18 on dist-test-slave-lkn4"
server_key: "aa943d13ec1fd7bd007130489b688693"
server_key_iv: "3b30946ae49ceaec2ffb7fb82d9230e8"
server_key_version: "encryptionkey@0"
I20260927 12:50:18.899266 20867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790513161160093-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790513161160093-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790513161160093-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:50:18.904106 20867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:50:18.907527 20867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:50:18.907727 20867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:50:18.914933 20867 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33037
I20260927 12:50:18.914944 20934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33037 every 8 connection(s)
I20260927 12:50:18.915910 20867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:50:18.918902 20935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:50:18.924881 20935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae4eab09c731424ab0961352468cde79: Bootstrap starting.
I20260927 12:50:18.926184 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 20867
I20260927 12:50:18.926306 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790513161160093-392-0/minicluster-data/master-0/wal/instance
I20260927 12:50:18.926568 392 external_mini_cluster.cc:1428] Setting key 80be1739c635fd972a5b1a62b142acb9
I20260927 12:50:18.927819 20935 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae4eab09c731424ab0961352468cde79: Neither blocks nor log segments found. Creating new log.
I20260927 12:50:18.928555 20935 log.cc:824] T 00000000000000000000000000000000 P ae4eab09c731424ab0961352468cde79: Log is configured to *not* fsync() on all Append() calls
I20260927 12:50:18.930827 20935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae4eab09c731424ab0961352468cde79: No bootstrap required, opened a new log
I20260927 12:50:18.931932 20938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:18.927735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32895 (local address 127.0.0.1:33037)
0927 12:50:18.928179 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:18.928195 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:18.928230 (+ 35us) server_negotiation.cc:407] Connection header received
0927 12:50:18.929024 (+ 794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:18.929050 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:18.929417 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:18.929740 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:18.930628 (+ 888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:18.930633 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:18.930645 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:50:18.930655 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:18.930772 (+ 117us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:50:18.930794 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:18.930887 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:18.931030 (+ 143us) server_negotiation.cc:299] Negotiation successful
0927 12:50:18.931182 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":128,"threads_started":1}
I20260927 12:50:18.934948 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:50:18.934866 20935 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae4eab09c731424ab0961352468cde79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae4eab09c731424ab0961352468cde79" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33037 } }
I20260927 12:50:18.935129 20935 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae4eab09c731424ab0961352468cde79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:50:18.935184 20935 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae4eab09c731424ab0961352468cde79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae4eab09c731424ab0961352468cde79, State: Initialized, Role: FOLLOWER
I20260927 12:50:18.935603 20935 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae4eab09c731424ab0961352468cde79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae4eab09c731424ab0961352468cde79" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33037 } }
I20260927 12:50:18.935720 20935 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae4eab09c731424ab0961352468cde79 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:50:18.935758 20935 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae4eab09c731424ab0961352468cde79 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:50:18.935858 20935 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae4eab09c731424ab0961352468cde79 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:50:18.936810 20935 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae4eab09c731424ab0961352468cde79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae4eab09c731424ab0961352468cde79" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33037 } }
I20260927 12:50:18.937161 20935 leader_election.cc:302] T 00000000000000000000000000000000 P ae4eab09c731424ab0961352468cde79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae4eab09c731424ab0961352468cde79; no voters:
I20260927 12:50:18.937453 20935 leader_election.cc:288] T 00000000000000000000000000000000 P ae4eab09c731424ab0961352468cde79 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:50:18.937533 20945 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae4eab09c731424ab0961352468cde79 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:50:18.937700 20945 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae4eab09c731424ab0961352468cde79 [term 1 LEADER]: Becoming Leader. State: Replica: ae4eab09c731424ab0961352468cde79, State: Running, Role: LEADER
I20260927 12:50:18.938072 20945 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae4eab09c731424ab0961352468cde79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae4eab09c731424ab0961352468cde79" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33037 } }
I20260927 12:50:18.938786 20935 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae4eab09c731424ab0961352468cde79 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:50:18.938961 20938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:18.937017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58573 (local address 127.0.0.1:33037)
0927 12:50:18.937142 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:18.937147 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:18.937427 (+ 280us) server_negotiation.cc:407] Connection header received
0927 12:50:18.937601 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:18.937605 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:18.937652 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:18.937701 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:18.938541 (+ 840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:18.938545 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:18.938548 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:50:18.938559 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:18.938582 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:50:18.938592 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:18.938676 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:18.938792 (+ 116us) server_negotiation.cc:299] Negotiation successful
0927 12:50:18.938833 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 12:50:18.939921 20947 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae4eab09c731424ab0961352468cde79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae4eab09c731424ab0961352468cde79. Latest consensus state: current_term: 1 leader_uuid: "ae4eab09c731424ab0961352468cde79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae4eab09c731424ab0961352468cde79" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33037 } } }
I20260927 12:50:18.940042 20947 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae4eab09c731424ab0961352468cde79 [sys.catalog]: This master's current role is: LEADER
I20260927 12:50:18.940339 20946 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae4eab09c731424ab0961352468cde79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae4eab09c731424ab0961352468cde79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae4eab09c731424ab0961352468cde79" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33037 } } }
I20260927 12:50:18.940392 20954 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:50:18.940444 20946 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae4eab09c731424ab0961352468cde79 [sys.catalog]: This master's current role is: LEADER
I20260927 12:50:18.943218 20954 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:50:18.949437 20954 catalog_manager.cc:1347] Generated new cluster ID: f8f51ceba1254a8f81e8e6e896beeae7
I20260927 12:50:18.949527 20954 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:50:18.983634 20954 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:50:18.984035 20954 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:50:18.991981 20954 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae4eab09c731424ab0961352468cde79: Generated new TSK 0
I20260927 12:50:18.992856 20954 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:50:18.998584 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 20867
2026-09-27T12:50:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1337 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790513161160093-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:50:19 dist-test-slave-lkn4 krb5kdc[20966](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:50:19 dist-test-slave-lkn4 krb5kdc[20966](info): set up 2 sockets
Sep 27 12:50:19 dist-test-slave-lkn4 krb5kdc[20966](info): commencing operation
krb5kdc: starting...
W20260927 12:50:19.954234 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790513161160093-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790513161160093-392-0/krb5kdc/test-user.keytab.
Sep 27 12:50:20 dist-test-slave-lkn4 krb5kdc[20966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:50:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:50:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790513161160093-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790513161160093-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 12:50:20.076560 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790513161160093-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790513161160093-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790513161160093-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32969
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790513161160093-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32969
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:50:20.186969 20979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:50:20.187268 20979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:50:20.187362 20979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:50:20.191452 20979 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:50:20.191522 20979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:50:20.191550 20979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:50:20.191570 20979 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:50:20.191592 20979 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:50:20.197009 20979 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790513161160093-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790513161160093-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790513161160093-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790513161160093-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:50:20.198132 20979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:50:20.199164 20979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:50:20.208070 20984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:50:20.208030 20985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:50:20.208052 20987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:50:20.208170 20979 server_base.cc:1034] running on GCE node
I20260927 12:50:20.208868 20979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:50:20.209971 20979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:50:20.211175 20979 hybrid_clock.cc:648] HybridClock initialized: now 1790513420211140 us; error 49 us; skew 500 ppm
Sep 27 12:50:20 dist-test-slave-lkn4 krb5kdc[20966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:50:20.219069 20979 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 12:50:20.220132 20979 webserver.cc:460] Webserver started at http://127.0.0.1:41367/ using document root <none> and password file <none>
I20260927 12:50:20.220768 20979 fs_manager.cc:359] Metadata directory not provided
I20260927 12:50:20.220842 20979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:50:20.221055 20979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:50:20.223311 20979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790513161160093-392-0/minicluster-data/master-0/data/instance:
uuid: "b004cd9f2ca24b6fad9ab19e12026f8c"
format_stamp: "Formatted at 2026-09-27 12:50:20 on dist-test-slave-lkn4"
server_key: "b14fd6af4d7bc7288ffee6ce3925e9c2"
server_key_iv: "c150a636262607e92389e5a6b0964dc4"
server_key_version: "encryptionkey@0"
I20260927 12:50:20.223848 20979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790513161160093-392-0/minicluster-data/master-0/wal/instance:
uuid: "b004cd9f2ca24b6fad9ab19e12026f8c"
format_stamp: "Formatted at 2026-09-27 12:50:20 on dist-test-slave-lkn4"
server_key: "b14fd6af4d7bc7288ffee6ce3925e9c2"
server_key_iv: "c150a636262607e92389e5a6b0964dc4"
server_key_version: "encryptionkey@0"
I20260927 12:50:20.227726 20979 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 12:50:20.230228 20994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:20.231532 20979 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:50:20.231757 20979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790513161160093-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790513161160093-392-0/minicluster-data/master-0/wal
uuid: "b004cd9f2ca24b6fad9ab19e12026f8c"
format_stamp: "Formatted at 2026-09-27 12:50:20 on dist-test-slave-lkn4"
server_key: "b14fd6af4d7bc7288ffee6ce3925e9c2"
server_key_iv: "c150a636262607e92389e5a6b0964dc4"
server_key_version: "encryptionkey@0"
I20260927 12:50:20.231886 20979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790513161160093-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790513161160093-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790513161160093-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:50:20.256491 20979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:50:20.259502 20979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:50:20.259696 20979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:50:20.266916 20979 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32969
I20260927 12:50:20.266976 21046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32969 every 8 connection(s)
I20260927 12:50:20.267927 20979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:50:20.270916 21047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:50:20.272989 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 20979
I20260927 12:50:20.273100 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790513161160093-392-0/minicluster-data/master-0/wal/instance
I20260927 12:50:20.273360 392 external_mini_cluster.cc:1428] Setting key 9b65fc856751ed02a5d4cce4130fc3e8
I20260927 12:50:20.277346 21047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b004cd9f2ca24b6fad9ab19e12026f8c: Bootstrap starting.
I20260927 12:50:20.279996 21047 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b004cd9f2ca24b6fad9ab19e12026f8c: Neither blocks nor log segments found. Creating new log.
I20260927 12:50:20.280869 21047 log.cc:824] T 00000000000000000000000000000000 P b004cd9f2ca24b6fad9ab19e12026f8c: Log is configured to *not* fsync() on all Append() calls
Sep 27 12:50:20 dist-test-slave-lkn4 krb5kdc[20966](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 12:50:20.283013 21047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b004cd9f2ca24b6fad9ab19e12026f8c: No bootstrap required, opened a new log
I20260927 12:50:20.287714 21047 raft_consensus.cc:348] T 00000000000000000000000000000000 P b004cd9f2ca24b6fad9ab19e12026f8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b004cd9f2ca24b6fad9ab19e12026f8c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32969 } }
I20260927 12:50:20.288028 21047 raft_consensus.cc:374] T 00000000000000000000000000000000 P b004cd9f2ca24b6fad9ab19e12026f8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:50:20.288084 21047 raft_consensus.cc:729] T 00000000000000000000000000000000 P b004cd9f2ca24b6fad9ab19e12026f8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b004cd9f2ca24b6fad9ab19e12026f8c, State: Initialized, Role: FOLLOWER
I20260927 12:50:20.288581 21047 consensus_queue.cc:260] T 00000000000000000000000000000000 P b004cd9f2ca24b6fad9ab19e12026f8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b004cd9f2ca24b6fad9ab19e12026f8c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32969 } }
I20260927 12:50:20.288753 21047 raft_consensus.cc:388] T 00000000000000000000000000000000 P b004cd9f2ca24b6fad9ab19e12026f8c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:50:20.288820 21047 raft_consensus.cc:482] T 00000000000000000000000000000000 P b004cd9f2ca24b6fad9ab19e12026f8c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:50:20.288913 21047 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b004cd9f2ca24b6fad9ab19e12026f8c [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:50:20.289043 21050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:20.274793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58509 (local address 127.0.0.1:32969)
0927 12:50:20.275232 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:20.275252 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:20.275296 (+ 44us) server_negotiation.cc:407] Connection header received
0927 12:50:20.276212 (+ 916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:20.276237 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:20.276700 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:20.277065 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:20.277749 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:20.279243 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:20.280105 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:20.280796 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:20.280956 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:20.283562 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:20.283568 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:20.283585 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:50:20.283616 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:20.285715 (+ 2099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:20.286188 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:20.286194 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:20.286200 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:20.286354 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:20.286655 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:20.286667 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:20.286669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:20.287224 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:20.287441 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:20.287708 (+ 267us) server_negotiation.cc:299] Negotiation successful
0927 12:50:20.288020 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":119,"threads_started":1}
I20260927 12:50:20.289942 21047 raft_consensus.cc:504] T 00000000000000000000000000000000 P b004cd9f2ca24b6fad9ab19e12026f8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b004cd9f2ca24b6fad9ab19e12026f8c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32969 } }
I20260927 12:50:20.290301 21047 leader_election.cc:302] T 00000000000000000000000000000000 P b004cd9f2ca24b6fad9ab19e12026f8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b004cd9f2ca24b6fad9ab19e12026f8c; no voters:
I20260927 12:50:20.290602 21047 leader_election.cc:288] T 00000000000000000000000000000000 P b004cd9f2ca24b6fad9ab19e12026f8c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:50:20.290716 21052 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b004cd9f2ca24b6fad9ab19e12026f8c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:50:20.290916 21052 raft_consensus.cc:686] T 00000000000000000000000000000000 P b004cd9f2ca24b6fad9ab19e12026f8c [term 1 LEADER]: Becoming Leader. State: Replica: b004cd9f2ca24b6fad9ab19e12026f8c, State: Running, Role: LEADER
I20260927 12:50:20.291301 21052 consensus_queue.cc:237] T 00000000000000000000000000000000 P b004cd9f2ca24b6fad9ab19e12026f8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b004cd9f2ca24b6fad9ab19e12026f8c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32969 } }
I20260927 12:50:20.291882 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:50:20.291957 21047 sys_catalog.cc:564] T 00000000000000000000000000000000 P b004cd9f2ca24b6fad9ab19e12026f8c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:50:20.292567 21052 sys_catalog.cc:455] T 00000000000000000000000000000000 P b004cd9f2ca24b6fad9ab19e12026f8c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b004cd9f2ca24b6fad9ab19e12026f8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b004cd9f2ca24b6fad9ab19e12026f8c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32969 } } }
I20260927 12:50:20.292742 21052 sys_catalog.cc:458] T 00000000000000000000000000000000 P b004cd9f2ca24b6fad9ab19e12026f8c [sys.catalog]: This master's current role is: LEADER
I20260927 12:50:20.292972 21052 sys_catalog.cc:455] T 00000000000000000000000000000000 P b004cd9f2ca24b6fad9ab19e12026f8c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b004cd9f2ca24b6fad9ab19e12026f8c. Latest consensus state: current_term: 1 leader_uuid: "b004cd9f2ca24b6fad9ab19e12026f8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b004cd9f2ca24b6fad9ab19e12026f8c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32969 } } }
I20260927 12:50:20.293103 21052 sys_catalog.cc:458] T 00000000000000000000000000000000 P b004cd9f2ca24b6fad9ab19e12026f8c [sys.catalog]: This master's current role is: LEADER
I20260927 12:50:20.293536 21059 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:50:20.297432 21059 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:50:20.302627 21050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:20.294131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33017 (local address 127.0.0.1:32969)
0927 12:50:20.294246 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:20.294251 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:20.294717 (+ 466us) server_negotiation.cc:407] Connection header received
0927 12:50:20.294826 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:20.294829 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:20.294872 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:20.294937 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:20.295578 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:20.296088 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:20.297190 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:20.297559 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:20.297652 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:20.299291 (+ 1639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:20.299295 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:20.299298 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:50:20.299323 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:20.300911 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:20.301398 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:20.301402 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:20.301404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:20.301491 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:20.301811 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:20.301815 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:20.301817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:20.301999 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:20.302142 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:20.302369 (+ 227us) server_negotiation.cc:299] Negotiation successful
0927 12:50:20.302477 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 12:50:20.304078 21059 catalog_manager.cc:1347] Generated new cluster ID: e8d390ef60024956831eeb852e5c3a7c
I20260927 12:50:20.304134 21059 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:50:20.332615 21059 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:50:20.333679 21059 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:50:20.341040 21059 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b004cd9f2ca24b6fad9ab19e12026f8c: Generated new TSK 0
I20260927 12:50:20.341820 21059 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:50:20.354013 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 20979
2026-09-27T12:50:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1355 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790513161160093-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:50:20 dist-test-slave-lkn4 krb5kdc[21078](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:50:20 dist-test-slave-lkn4 krb5kdc[21078](info): set up 2 sockets
Sep 27 12:50:20 dist-test-slave-lkn4 krb5kdc[21078](info): commencing operation
krb5kdc: starting...
W20260927 12:50:21.310137 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790513161160093-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790513161160093-392-0/krb5kdc/test-user.keytab.
Sep 27 12:50:21 dist-test-slave-lkn4 krb5kdc[21078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:50:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:50:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790513161160093-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790513161160093-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 12:50:21.432940 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790513161160093-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790513161160093-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790513161160093-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45599
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790513161160093-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45599
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:50:21.544287 21091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:50:21.544600 21091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:50:21.544661 21091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:50:21.548821 21091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:50:21.548900 21091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:50:21.548923 21091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:50:21.548944 21091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:50:21.548979 21091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:50:21.554574 21091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790513161160093-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790513161160093-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45599
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790513161160093-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790513161160093-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:50:21.555723 21091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:50:21.556773 21091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:50:21.565315 21096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:50:21.565348 21097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:50:21.565348 21099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:50:21.565706 21091 server_base.cc:1034] running on GCE node
I20260927 12:50:21.566408 21091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:50:21.567695 21091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:50:21.568889 21091 hybrid_clock.cc:648] HybridClock initialized: now 1790513421568878 us; error 37 us; skew 500 ppm
Sep 27 12:50:21 dist-test-slave-lkn4 krb5kdc[21078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513421, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:50:21.571898 21091 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 12:50:21.572845 21091 webserver.cc:460] Webserver started at http://127.0.0.1:35933/ using document root <none> and password file <none>
I20260927 12:50:21.573390 21091 fs_manager.cc:359] Metadata directory not provided
I20260927 12:50:21.573458 21091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:50:21.573675 21091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:50:21.575922 21091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790513161160093-392-0/minicluster-data/master-0/data/instance:
uuid: "7d3e5f41f5e24c3ea7c24a7adab56a2b"
format_stamp: "Formatted at 2026-09-27 12:50:21 on dist-test-slave-lkn4"
server_key: "92944c723822c4ea46e428411fa96822"
server_key_iv: "48fa302c8f9d19109c0d8e00de1c701c"
server_key_version: "encryptionkey@0"
I20260927 12:50:21.576551 21091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790513161160093-392-0/minicluster-data/master-0/wal/instance:
uuid: "7d3e5f41f5e24c3ea7c24a7adab56a2b"
format_stamp: "Formatted at 2026-09-27 12:50:21 on dist-test-slave-lkn4"
server_key: "92944c723822c4ea46e428411fa96822"
server_key_iv: "48fa302c8f9d19109c0d8e00de1c701c"
server_key_version: "encryptionkey@0"
I20260927 12:50:21.580432 21091 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260927 12:50:21.582957 21106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:21.584174 21091 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:50:21.584292 21091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790513161160093-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790513161160093-392-0/minicluster-data/master-0/wal
uuid: "7d3e5f41f5e24c3ea7c24a7adab56a2b"
format_stamp: "Formatted at 2026-09-27 12:50:21 on dist-test-slave-lkn4"
server_key: "92944c723822c4ea46e428411fa96822"
server_key_iv: "48fa302c8f9d19109c0d8e00de1c701c"
server_key_version: "encryptionkey@0"
I20260927 12:50:21.584396 21091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790513161160093-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790513161160093-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790513161160093-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:50:21.598229 21091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:50:21.601302 21091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:50:21.601490 21091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:50:21.608355 21091 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45599
I20260927 12:50:21.608373 21158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45599 every 8 connection(s)
I20260927 12:50:21.609295 21091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:50:21.612174 21159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:50:21.617834 21159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d3e5f41f5e24c3ea7c24a7adab56a2b: Bootstrap starting.
I20260927 12:50:21.619580 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 21091
I20260927 12:50:21.619724 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790513161160093-392-0/minicluster-data/master-0/wal/instance
I20260927 12:50:21.620020 392 external_mini_cluster.cc:1428] Setting key b8be66581208eec06cce026b35834208
I20260927 12:50:21.620585 21159 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d3e5f41f5e24c3ea7c24a7adab56a2b: Neither blocks nor log segments found. Creating new log.
I20260927 12:50:21.621464 21159 log.cc:824] T 00000000000000000000000000000000 P 7d3e5f41f5e24c3ea7c24a7adab56a2b: Log is configured to *not* fsync() on all Append() calls
I20260927 12:50:21.623510 21159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d3e5f41f5e24c3ea7c24a7adab56a2b: No bootstrap required, opened a new log
I20260927 12:50:21.627009 21159 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d3e5f41f5e24c3ea7c24a7adab56a2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d3e5f41f5e24c3ea7c24a7adab56a2b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45599 } }
I20260927 12:50:21.627235 21159 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d3e5f41f5e24c3ea7c24a7adab56a2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:50:21.627270 21159 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d3e5f41f5e24c3ea7c24a7adab56a2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d3e5f41f5e24c3ea7c24a7adab56a2b, State: Initialized, Role: FOLLOWER
I20260927 12:50:21.627748 21159 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d3e5f41f5e24c3ea7c24a7adab56a2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d3e5f41f5e24c3ea7c24a7adab56a2b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45599 } }
I20260927 12:50:21.627887 21159 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d3e5f41f5e24c3ea7c24a7adab56a2b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:50:21.627949 21159 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d3e5f41f5e24c3ea7c24a7adab56a2b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:50:21.628041 21159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d3e5f41f5e24c3ea7c24a7adab56a2b [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:50:21.628652 21162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:21.621148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43313 (local address 127.0.0.1:45599)
0927 12:50:21.621600 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:21.621620 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:21.621656 (+ 36us) server_negotiation.cc:407] Connection header received
0927 12:50:21.622492 (+ 836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:21.622514 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:21.622867 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:21.623175 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:21.623572 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:21.624537 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:21.625391 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:21.625864 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:21.626007 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:21.626832 (+ 825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:21.626837 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:21.626850 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:50:21.627223 (+ 373us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:21.627293 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:50:21.627318 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:21.627406 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:21.627531 (+ 125us) server_negotiation.cc:299] Negotiation successful
0927 12:50:21.627731 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":148,"threads_started":1}
I20260927 12:50:21.629115 21159 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d3e5f41f5e24c3ea7c24a7adab56a2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d3e5f41f5e24c3ea7c24a7adab56a2b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45599 } }
I20260927 12:50:21.629465 21159 leader_election.cc:302] T 00000000000000000000000000000000 P 7d3e5f41f5e24c3ea7c24a7adab56a2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d3e5f41f5e24c3ea7c24a7adab56a2b; no voters:
I20260927 12:50:21.629799 21159 leader_election.cc:288] T 00000000000000000000000000000000 P 7d3e5f41f5e24c3ea7c24a7adab56a2b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:50:21.629863 21164 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d3e5f41f5e24c3ea7c24a7adab56a2b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:50:21.630082 21164 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d3e5f41f5e24c3ea7c24a7adab56a2b [term 1 LEADER]: Becoming Leader. State: Replica: 7d3e5f41f5e24c3ea7c24a7adab56a2b, State: Running, Role: LEADER
I20260927 12:50:21.630381 21164 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d3e5f41f5e24c3ea7c24a7adab56a2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d3e5f41f5e24c3ea7c24a7adab56a2b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45599 } }
I20260927 12:50:21.631058 21159 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d3e5f41f5e24c3ea7c24a7adab56a2b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:50:21.631598 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:50:21.631814 21166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d3e5f41f5e24c3ea7c24a7adab56a2b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d3e5f41f5e24c3ea7c24a7adab56a2b. Latest consensus state: current_term: 1 leader_uuid: "7d3e5f41f5e24c3ea7c24a7adab56a2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d3e5f41f5e24c3ea7c24a7adab56a2b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45599 } } }
I20260927 12:50:21.631932 21166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d3e5f41f5e24c3ea7c24a7adab56a2b [sys.catalog]: This master's current role is: LEADER
I20260927 12:50:21.631909 21165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d3e5f41f5e24c3ea7c24a7adab56a2b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d3e5f41f5e24c3ea7c24a7adab56a2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d3e5f41f5e24c3ea7c24a7adab56a2b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45599 } } }
I20260927 12:50:21.631994 21165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d3e5f41f5e24c3ea7c24a7adab56a2b [sys.catalog]: This master's current role is: LEADER
I20260927 12:50:21.632272 21173 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:50:21.635337 21173 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:50:21.638760 21162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:21.633771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34083 (local address 127.0.0.1:45599)
0927 12:50:21.633866 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:21.633870 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:21.634384 (+ 514us) server_negotiation.cc:407] Connection header received
0927 12:50:21.634507 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:21.634510 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:21.634567 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:21.634644 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:21.634918 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:21.635670 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:21.636965 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:21.637320 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:21.637425 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:21.638319 (+ 894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:21.638323 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:21.638326 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:50:21.638342 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:21.638371 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:50:21.638385 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:21.638462 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:21.638549 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 12:50:21.638610 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260927 12:50:21.641369 21173 catalog_manager.cc:1347] Generated new cluster ID: b9d8cec337e744baaef9801a0cdd9de9
I20260927 12:50:21.641441 21173 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:50:21.655081 21173 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:50:21.656051 21173 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:50:21.662786 21173 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d3e5f41f5e24c3ea7c24a7adab56a2b: Generated new TSK 0
I20260927 12:50:21.663486 21173 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:50:21.673025 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 21091
2026-09-27T12:50:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1318 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790513161160093-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:50:21 dist-test-slave-lkn4 krb5kdc[21190](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:50:21 dist-test-slave-lkn4 krb5kdc[21190](info): set up 2 sockets
Sep 27 12:50:21 dist-test-slave-lkn4 krb5kdc[21190](info): commencing operation
krb5kdc: starting...
W20260927 12:50:22.634024 392 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790513161160093-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790513161160093-392-0/krb5kdc/test-user.keytab.
Sep 27 12:50:22 dist-test-slave-lkn4 krb5kdc[21190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:50:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:50:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790513161160093-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790513161160093-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 12:50:22.762530 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790513161160093-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790513161160093-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790513161160093-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34047
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790513161160093-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34047
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:50:22.875185 21203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:50:22.875488 21203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:50:22.875548 21203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:50:22.879664 21203 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:50:22.879742 21203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:50:22.879769 21203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:50:22.879791 21203 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:50:22.879817 21203 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:50:22.885408 21203 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790513161160093-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790513161160093-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34047
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790513161160093-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21203
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790513161160093-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:50:22.886693 21203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:50:22.887786 21203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:50:22.896793 21208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:50:22.896853 21203 server_base.cc:1034] running on GCE node
W20260927 12:50:22.896821 21211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:50:22.897032 21209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:50:22.897629 21203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:50:22.898861 21203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:50:22.900048 21203 hybrid_clock.cc:648] HybridClock initialized: now 1790513422900029 us; error 39 us; skew 500 ppm
Sep 27 12:50:22 dist-test-slave-lkn4 krb5kdc[21190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513422, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:50:22.903105 21203 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 12:50:22.904063 21203 webserver.cc:460] Webserver started at http://127.0.0.1:36633/ using document root <none> and password file <none>
I20260927 12:50:22.904608 21203 fs_manager.cc:359] Metadata directory not provided
I20260927 12:50:22.904667 21203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:50:22.904846 21203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:50:22.907002 21203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790513161160093-392-0/minicluster-data/master-0/data/instance:
uuid: "5d93ab58cbd1417887e4cc98c3757b58"
format_stamp: "Formatted at 2026-09-27 12:50:22 on dist-test-slave-lkn4"
server_key: "e4b7bc7bdeb4e736a517cf98843bb4fe"
server_key_iv: "d57145f6055e6341ef969200091192f5"
server_key_version: "encryptionkey@0"
I20260927 12:50:22.907519 21203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790513161160093-392-0/minicluster-data/master-0/wal/instance:
uuid: "5d93ab58cbd1417887e4cc98c3757b58"
format_stamp: "Formatted at 2026-09-27 12:50:22 on dist-test-slave-lkn4"
server_key: "e4b7bc7bdeb4e736a517cf98843bb4fe"
server_key_iv: "d57145f6055e6341ef969200091192f5"
server_key_version: "encryptionkey@0"
I20260927 12:50:22.911317 21203 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260927 12:50:22.913825 21218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:22.915236 21203 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:50:22.915365 21203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790513161160093-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790513161160093-392-0/minicluster-data/master-0/wal
uuid: "5d93ab58cbd1417887e4cc98c3757b58"
format_stamp: "Formatted at 2026-09-27 12:50:22 on dist-test-slave-lkn4"
server_key: "e4b7bc7bdeb4e736a517cf98843bb4fe"
server_key_iv: "d57145f6055e6341ef969200091192f5"
server_key_version: "encryptionkey@0"
I20260927 12:50:22.915473 21203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790513161160093-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790513161160093-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790513161160093-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:50:22.920495 21203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:50:22.923911 21203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:50:22.924114 21203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:50:22.931849 21203 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34047
I20260927 12:50:22.931857 21270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34047 every 8 connection(s)
I20260927 12:50:22.932928 21203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:50:22.935881 21271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:50:22.939071 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 21203
I20260927 12:50:22.939205 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790513161160093-392-0/minicluster-data/master-0/wal/instance
I20260927 12:50:22.939492 392 external_mini_cluster.cc:1428] Setting key ce9d9651f49ecd1c8f3de5b2ae119ed4
I20260927 12:50:22.942066 21271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d93ab58cbd1417887e4cc98c3757b58: Bootstrap starting.
I20260927 12:50:22.944479 21271 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d93ab58cbd1417887e4cc98c3757b58: Neither blocks nor log segments found. Creating new log.
I20260927 12:50:22.945080 21274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:22.940951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53619 (local address 127.0.0.1:34047)
0927 12:50:22.941413 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:22.941429 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:22.941467 (+ 38us) server_negotiation.cc:407] Connection header received
0927 12:50:22.942226 (+ 759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:22.942246 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:22.942559 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:22.942877 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:22.943694 (+ 817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:22.943699 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:22.943711 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:50:22.943721 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:22.943793 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:50:22.943814 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:22.943908 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:22.944051 (+ 143us) server_negotiation.cc:299] Negotiation successful
0927 12:50:22.944208 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":140,"threads_started":1}
I20260927 12:50:22.945235 21271 log.cc:824] T 00000000000000000000000000000000 P 5d93ab58cbd1417887e4cc98c3757b58: Log is configured to *not* fsync() on all Append() calls
I20260927 12:50:22.948143 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:50:22.948282 21271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d93ab58cbd1417887e4cc98c3757b58: No bootstrap required, opened a new log
I20260927 12:50:22.952204 21274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:22.950351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58687 (local address 127.0.0.1:34047)
0927 12:50:22.950468 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:22.950473 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:22.950666 (+ 193us) server_negotiation.cc:407] Connection header received
0927 12:50:22.950796 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:22.950799 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:22.950836 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:22.950902 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:22.951638 (+ 736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:22.951642 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:22.951645 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:50:22.951658 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:22.951687 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:50:22.951701 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:22.951782 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:22.952063 (+ 281us) server_negotiation.cc:299] Negotiation successful
0927 12:50:22.952100 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 12:50:22.952530 21271 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d93ab58cbd1417887e4cc98c3757b58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d93ab58cbd1417887e4cc98c3757b58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34047 } }
I20260927 12:50:22.952869 21271 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d93ab58cbd1417887e4cc98c3757b58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:50:22.952939 21271 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d93ab58cbd1417887e4cc98c3757b58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d93ab58cbd1417887e4cc98c3757b58, State: Initialized, Role: FOLLOWER
I20260927 12:50:22.953465 21271 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d93ab58cbd1417887e4cc98c3757b58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d93ab58cbd1417887e4cc98c3757b58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34047 } }
I20260927 12:50:22.953616 21271 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d93ab58cbd1417887e4cc98c3757b58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:50:22.953667 21271 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d93ab58cbd1417887e4cc98c3757b58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:50:22.953744 21271 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d93ab58cbd1417887e4cc98c3757b58 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:50:22.954744 21271 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d93ab58cbd1417887e4cc98c3757b58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d93ab58cbd1417887e4cc98c3757b58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34047 } }
I20260927 12:50:22.955080 21271 leader_election.cc:302] T 00000000000000000000000000000000 P 5d93ab58cbd1417887e4cc98c3757b58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d93ab58cbd1417887e4cc98c3757b58; no voters:
I20260927 12:50:22.955363 21271 leader_election.cc:288] T 00000000000000000000000000000000 P 5d93ab58cbd1417887e4cc98c3757b58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:50:22.955543 21281 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d93ab58cbd1417887e4cc98c3757b58 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:50:22.955746 21281 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d93ab58cbd1417887e4cc98c3757b58 [term 1 LEADER]: Becoming Leader. State: Replica: 5d93ab58cbd1417887e4cc98c3757b58, State: Running, Role: LEADER
I20260927 12:50:22.956055 21281 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d93ab58cbd1417887e4cc98c3757b58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d93ab58cbd1417887e4cc98c3757b58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34047 } }
I20260927 12:50:22.956672 21271 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d93ab58cbd1417887e4cc98c3757b58 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:50:22.957731 21282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d93ab58cbd1417887e4cc98c3757b58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d93ab58cbd1417887e4cc98c3757b58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d93ab58cbd1417887e4cc98c3757b58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34047 } } }
I20260927 12:50:22.957849 21282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d93ab58cbd1417887e4cc98c3757b58 [sys.catalog]: This master's current role is: LEADER
I20260927 12:50:22.958173 21283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d93ab58cbd1417887e4cc98c3757b58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d93ab58cbd1417887e4cc98c3757b58. Latest consensus state: current_term: 1 leader_uuid: "5d93ab58cbd1417887e4cc98c3757b58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d93ab58cbd1417887e4cc98c3757b58" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34047 } } }
I20260927 12:50:22.958338 21283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d93ab58cbd1417887e4cc98c3757b58 [sys.catalog]: This master's current role is: LEADER
I20260927 12:50:22.958253 21290 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:50:22.961831 21290 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:50:22.968123 21290 catalog_manager.cc:1347] Generated new cluster ID: 67114efceaa945b6b857b652d7765a45
I20260927 12:50:22.968225 21290 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:50:22.976677 21290 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:50:22.977049 21290 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:50:22.986925 21290 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d93ab58cbd1417887e4cc98c3757b58: Generated new TSK 0
I20260927 12:50:22.987831 21290 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:50:23.001436 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 21203
2026-09-27T12:50:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1328 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790513161160093-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:50:23 dist-test-slave-lkn4 krb5kdc[21302](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:50:23 dist-test-slave-lkn4 krb5kdc[21302](info): set up 2 sockets
Sep 27 12:50:23 dist-test-slave-lkn4 krb5kdc[21302](info): commencing operation
krb5kdc: starting...
W20260927 12:50:23.958133 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790513161160093-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790513161160093-392-0/krb5kdc/test-user.keytab.
Sep 27 12:50:24 dist-test-slave-lkn4 krb5kdc[21302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:50:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:50:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790513161160093-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790513161160093-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 12:50:24.075297 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790513161160093-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790513161160093-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790513161160093-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46341
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790513161160093-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46341
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:50:24.185068 21315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:50:24.185379 21315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:50:24.185422 21315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:50:24.189523 21315 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:50:24.189605 21315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:50:24.189623 21315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:50:24.189638 21315 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:50:24.189661 21315 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:50:24.195247 21315 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790513161160093-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790513161160093-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46341
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790513161160093-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790513161160093-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:50:24.196310 21315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:50:24.197315 21315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:50:24.205746 21320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:50:24.205822 21315 server_base.cc:1034] running on GCE node
W20260927 12:50:24.205786 21323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:50:24.205745 21321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:50:24.206568 21315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:50:24.207645 21315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:50:24.208827 21315 hybrid_clock.cc:648] HybridClock initialized: now 1790513424208808 us; error 31 us; skew 500 ppm
Sep 27 12:50:24 dist-test-slave-lkn4 krb5kdc[21302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513424, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:50:24.211585 21315 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 12:50:24.212419 21315 webserver.cc:460] Webserver started at http://127.0.0.1:42269/ using document root <none> and password file <none>
I20260927 12:50:24.212942 21315 fs_manager.cc:359] Metadata directory not provided
I20260927 12:50:24.212997 21315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:50:24.213155 21315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:50:24.215157 21315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790513161160093-392-0/minicluster-data/master-0/data/instance:
uuid: "9f3a858c070349ad992d26da942a9dbd"
format_stamp: "Formatted at 2026-09-27 12:50:24 on dist-test-slave-lkn4"
server_key: "c191d6ae860968a2c0162acdcf3e2a78"
server_key_iv: "eb5777a025ad442f4471e485c4e1eb24"
server_key_version: "encryptionkey@0"
I20260927 12:50:24.215663 21315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790513161160093-392-0/minicluster-data/master-0/wal/instance:
uuid: "9f3a858c070349ad992d26da942a9dbd"
format_stamp: "Formatted at 2026-09-27 12:50:24 on dist-test-slave-lkn4"
server_key: "c191d6ae860968a2c0162acdcf3e2a78"
server_key_iv: "eb5777a025ad442f4471e485c4e1eb24"
server_key_version: "encryptionkey@0"
I20260927 12:50:24.219266 21315 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 12:50:24.221673 21330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:24.222951 21315 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:50:24.223067 21315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790513161160093-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790513161160093-392-0/minicluster-data/master-0/wal
uuid: "9f3a858c070349ad992d26da942a9dbd"
format_stamp: "Formatted at 2026-09-27 12:50:24 on dist-test-slave-lkn4"
server_key: "c191d6ae860968a2c0162acdcf3e2a78"
server_key_iv: "eb5777a025ad442f4471e485c4e1eb24"
server_key_version: "encryptionkey@0"
I20260927 12:50:24.223167 21315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790513161160093-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790513161160093-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790513161160093-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:50:24.238642 21315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:50:24.241948 21315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:50:24.242187 21315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:50:24.249505 21315 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46341
I20260927 12:50:24.249552 21382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46341 every 8 connection(s)
I20260927 12:50:24.250517 21315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:50:24.251495 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 21315
I20260927 12:50:24.251657 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790513161160093-392-0/minicluster-data/master-0/wal/instance
I20260927 12:50:24.251943 392 external_mini_cluster.cc:1428] Setting key ebbbfc84ac234288ea3c00e7e5140052
I20260927 12:50:24.253808 21383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:50:24.259940 21383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f3a858c070349ad992d26da942a9dbd: Bootstrap starting.
Sep 27 12:50:24 dist-test-slave-lkn4 krb5kdc[21302](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 12:50:24.262454 21383 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f3a858c070349ad992d26da942a9dbd: Neither blocks nor log segments found. Creating new log.
I20260927 12:50:24.263151 21383 log.cc:824] T 00000000000000000000000000000000 P 9f3a858c070349ad992d26da942a9dbd: Log is configured to *not* fsync() on all Append() calls
I20260927 12:50:24.265136 21383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f3a858c070349ad992d26da942a9dbd: No bootstrap required, opened a new log
I20260927 12:50:24.266418 21386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:24.253291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.112:38583 (local address 127.0.0.1:46341)
0927 12:50:24.253711 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:24.253726 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:24.253761 (+ 35us) server_negotiation.cc:407] Connection header received
0927 12:50:24.254485 (+ 724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:24.254505 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:24.254933 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:24.255210 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:24.256062 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:24.257156 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:24.258254 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:24.258736 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:24.258851 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:24.261753 (+ 2902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:24.261759 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:24.261773 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:50:24.261800 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:24.263542 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:24.264082 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:24.264087 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:24.264091 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:24.264213 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:24.264516 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:24.264518 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:24.264519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:24.264964 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:24.265118 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:24.265433 (+ 315us) server_negotiation.cc:299] Negotiation successful
0927 12:50:24.265638 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":119,"threads_started":1}
I20260927 12:50:24.268673 21383 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f3a858c070349ad992d26da942a9dbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f3a858c070349ad992d26da942a9dbd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46341 } }
I20260927 12:50:24.268886 21383 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f3a858c070349ad992d26da942a9dbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:50:24.268934 21383 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f3a858c070349ad992d26da942a9dbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f3a858c070349ad992d26da942a9dbd, State: Initialized, Role: FOLLOWER
I20260927 12:50:24.269240 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:50:24.269373 21383 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f3a858c070349ad992d26da942a9dbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f3a858c070349ad992d26da942a9dbd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46341 } }
I20260927 12:50:24.269522 21383 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f3a858c070349ad992d26da942a9dbd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:50:24.269577 21383 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f3a858c070349ad992d26da942a9dbd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:50:24.269672 21383 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f3a858c070349ad992d26da942a9dbd [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:50:24.270871 21383 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f3a858c070349ad992d26da942a9dbd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f3a858c070349ad992d26da942a9dbd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46341 } }
I20260927 12:50:24.271207 21383 leader_election.cc:302] T 00000000000000000000000000000000 P 9f3a858c070349ad992d26da942a9dbd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f3a858c070349ad992d26da942a9dbd; no voters:
I20260927 12:50:24.271540 21383 leader_election.cc:288] T 00000000000000000000000000000000 P 9f3a858c070349ad992d26da942a9dbd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:50:24.271603 21393 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f3a858c070349ad992d26da942a9dbd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:50:24.271890 21393 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f3a858c070349ad992d26da942a9dbd [term 1 LEADER]: Becoming Leader. State: Replica: 9f3a858c070349ad992d26da942a9dbd, State: Running, Role: LEADER
I20260927 12:50:24.272238 21393 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f3a858c070349ad992d26da942a9dbd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f3a858c070349ad992d26da942a9dbd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46341 } }
I20260927 12:50:24.272917 21383 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f3a858c070349ad992d26da942a9dbd [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:50:24.273700 21394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f3a858c070349ad992d26da942a9dbd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f3a858c070349ad992d26da942a9dbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f3a858c070349ad992d26da942a9dbd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46341 } } }
I20260927 12:50:24.273819 21394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f3a858c070349ad992d26da942a9dbd [sys.catalog]: This master's current role is: LEADER
I20260927 12:50:24.274343 21395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f3a858c070349ad992d26da942a9dbd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f3a858c070349ad992d26da942a9dbd. Latest consensus state: current_term: 1 leader_uuid: "9f3a858c070349ad992d26da942a9dbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f3a858c070349ad992d26da942a9dbd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46341 } } }
I20260927 12:50:24.274478 21395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f3a858c070349ad992d26da942a9dbd [sys.catalog]: This master's current role is: LEADER
I20260927 12:50:24.275321 21402 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:50:24.278367 21402 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:50:24.279845 21386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:24.271360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.112:38261 (local address 127.0.0.1:46341)
0927 12:50:24.271471 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:24.271476 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:24.271709 (+ 233us) server_negotiation.cc:407] Connection header received
0927 12:50:24.271889 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:24.271892 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:24.271936 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:24.271981 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:24.272863 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:24.273653 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:24.274781 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:24.275088 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:24.275199 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:24.276935 (+ 1736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:24.276938 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:24.276940 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:50:24.276962 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:24.278266 (+ 1304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:24.278714 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:24.278717 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:24.278718 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:24.278788 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:24.279080 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:24.279082 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:24.279083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:24.279252 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:24.279377 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:24.279626 (+ 249us) server_negotiation.cc:299] Negotiation successful
0927 12:50:24.279712 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 12:50:24.284341 21402 catalog_manager.cc:1347] Generated new cluster ID: 3dc9bf9a31b740329799620994c59f73
I20260927 12:50:24.284406 21402 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:50:24.301503 21402 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:50:24.302919 21402 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:50:24.310029 21402 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f3a858c070349ad992d26da942a9dbd: Generated new TSK 0
I20260927 12:50:24.310917 21402 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:50:24.323204 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 21315
W20260927 12:50:24.327876 21300 connection.cc:535] client connection to 127.0.0.1:46341 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46341): Connection reset by peer (error 104)
2026-09-27T12:50:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1321 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790513161160093-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:50:24 dist-test-slave-lkn4 krb5kdc[21414](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:50:24 dist-test-slave-lkn4 krb5kdc[21414](info): set up 2 sockets
Sep 27 12:50:24 dist-test-slave-lkn4 krb5kdc[21414](info): commencing operation
krb5kdc: starting...
W20260927 12:50:25.294086 392 mini_kdc.cc:121] Time spent starting KDC: real 0.952s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790513161160093-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790513161160093-392-0/krb5kdc/test-user.keytab.
Sep 27 12:50:25 dist-test-slave-lkn4 krb5kdc[21414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513425, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:50:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:50:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790513161160093-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790513161160093-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 12:50:25.415095 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790513161160093-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790513161160093-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790513161160093-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34733
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790513161160093-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34733
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:50:25.525019 21427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:50:25.525322 21427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:50:25.525364 21427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:50:25.529932 21427 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:50:25.530077 21427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:50:25.530109 21427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:50:25.530148 21427 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:50:25.530170 21427 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:50:25.536139 21427 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790513161160093-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790513161160093-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790513161160093-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790513161160093-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:50:25.537257 21427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:50:25.538317 21427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:50:25.547295 21432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:50:25.547314 21433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:50:25.547410 21427 server_base.cc:1034] running on GCE node
W20260927 12:50:25.547294 21435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:50:25.548220 21427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:50:25.549386 21427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:50:25.550558 21427 hybrid_clock.cc:648] HybridClock initialized: now 1790513425550537 us; error 36 us; skew 500 ppm
Sep 27 12:50:25 dist-test-slave-lkn4 krb5kdc[21414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513425, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:50:25.557101 21427 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 12:50:25.558218 21427 webserver.cc:460] Webserver started at http://127.0.0.1:40263/ using document root <none> and password file <none>
I20260927 12:50:25.558789 21427 fs_manager.cc:359] Metadata directory not provided
I20260927 12:50:25.558854 21427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:50:25.559051 21427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:50:25.561255 21427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790513161160093-392-0/minicluster-data/master-0/data/instance:
uuid: "8404bf57902c4cd39efb499dea74e6eb"
format_stamp: "Formatted at 2026-09-27 12:50:25 on dist-test-slave-lkn4"
server_key: "6e65dbb04cdfce967575c64d4bc78df4"
server_key_iv: "829890cbe935972cb687355ad0419745"
server_key_version: "encryptionkey@0"
I20260927 12:50:25.561808 21427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790513161160093-392-0/minicluster-data/master-0/wal/instance:
uuid: "8404bf57902c4cd39efb499dea74e6eb"
format_stamp: "Formatted at 2026-09-27 12:50:25 on dist-test-slave-lkn4"
server_key: "6e65dbb04cdfce967575c64d4bc78df4"
server_key_iv: "829890cbe935972cb687355ad0419745"
server_key_version: "encryptionkey@0"
I20260927 12:50:25.565865 21427 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260927 12:50:25.568534 21442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:25.569896 21427 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 12:50:25.570006 21427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790513161160093-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790513161160093-392-0/minicluster-data/master-0/wal
uuid: "8404bf57902c4cd39efb499dea74e6eb"
format_stamp: "Formatted at 2026-09-27 12:50:25 on dist-test-slave-lkn4"
server_key: "6e65dbb04cdfce967575c64d4bc78df4"
server_key_iv: "829890cbe935972cb687355ad0419745"
server_key_version: "encryptionkey@0"
I20260927 12:50:25.570156 21427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790513161160093-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790513161160093-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790513161160093-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:50:25.584188 21427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:50:25.587554 21427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:50:25.587735 21427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:50:25.594892 21427 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34733
I20260927 12:50:25.594939 21494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34733 every 8 connection(s)
I20260927 12:50:25.595903 21427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:50:25.598907 21495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:50:25.601766 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 21427
I20260927 12:50:25.601902 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790513161160093-392-0/minicluster-data/master-0/wal/instance
I20260927 12:50:25.602185 392 external_mini_cluster.cc:1428] Setting key 444ff19a66f5e4bc5f5fec6761eda7de
I20260927 12:50:25.604880 21495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8404bf57902c4cd39efb499dea74e6eb: Bootstrap starting.
I20260927 12:50:25.607429 21495 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8404bf57902c4cd39efb499dea74e6eb: Neither blocks nor log segments found. Creating new log.
I20260927 12:50:25.608371 21495 log.cc:824] T 00000000000000000000000000000000 P 8404bf57902c4cd39efb499dea74e6eb: Log is configured to *not* fsync() on all Append() calls
I20260927 12:50:25.610711 21495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8404bf57902c4cd39efb499dea74e6eb: No bootstrap required, opened a new log
I20260927 12:50:25.611639 21498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:25.603638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.112:54379 (local address 127.0.0.1:34733)
0927 12:50:25.604090 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:25.604106 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:25.604143 (+ 37us) server_negotiation.cc:407] Connection header received
0927 12:50:25.604784 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:25.604804 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:25.605203 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:25.605506 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:25.605851 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:25.606889 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:25.608054 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:25.608580 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:25.608698 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:25.609605 (+ 907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:25.609611 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:25.609628 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:50:25.610143 (+ 515us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:25.610240 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:50:25.610279 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:25.610414 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:25.610566 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 12:50:25.610699 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":141,"threads_started":1}
I20260927 12:50:25.614523 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:50:25.614573 21495 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8404bf57902c4cd39efb499dea74e6eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8404bf57902c4cd39efb499dea74e6eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34733 } }
I20260927 12:50:25.614856 21495 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8404bf57902c4cd39efb499dea74e6eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:50:25.614907 21495 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8404bf57902c4cd39efb499dea74e6eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8404bf57902c4cd39efb499dea74e6eb, State: Initialized, Role: FOLLOWER
I20260927 12:50:25.615386 21495 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8404bf57902c4cd39efb499dea74e6eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8404bf57902c4cd39efb499dea74e6eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34733 } }
I20260927 12:50:25.615543 21495 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8404bf57902c4cd39efb499dea74e6eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:50:25.615607 21495 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8404bf57902c4cd39efb499dea74e6eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:50:25.615705 21495 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8404bf57902c4cd39efb499dea74e6eb [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:50:25.616853 21495 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8404bf57902c4cd39efb499dea74e6eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8404bf57902c4cd39efb499dea74e6eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34733 } }
I20260927 12:50:25.617226 21495 leader_election.cc:302] T 00000000000000000000000000000000 P 8404bf57902c4cd39efb499dea74e6eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8404bf57902c4cd39efb499dea74e6eb; no voters:
I20260927 12:50:25.617564 21495 leader_election.cc:288] T 00000000000000000000000000000000 P 8404bf57902c4cd39efb499dea74e6eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:50:25.617676 21505 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8404bf57902c4cd39efb499dea74e6eb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:50:25.617887 21505 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8404bf57902c4cd39efb499dea74e6eb [term 1 LEADER]: Becoming Leader. State: Replica: 8404bf57902c4cd39efb499dea74e6eb, State: Running, Role: LEADER
I20260927 12:50:25.618245 21505 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8404bf57902c4cd39efb499dea74e6eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8404bf57902c4cd39efb499dea74e6eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34733 } }
I20260927 12:50:25.618815 21495 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8404bf57902c4cd39efb499dea74e6eb [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:50:25.619865 21507 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8404bf57902c4cd39efb499dea74e6eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8404bf57902c4cd39efb499dea74e6eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8404bf57902c4cd39efb499dea74e6eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34733 } } }
I20260927 12:50:25.619992 21507 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8404bf57902c4cd39efb499dea74e6eb [sys.catalog]: This master's current role is: LEADER
I20260927 12:50:25.620246 21507 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8404bf57902c4cd39efb499dea74e6eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8404bf57902c4cd39efb499dea74e6eb. Latest consensus state: current_term: 1 leader_uuid: "8404bf57902c4cd39efb499dea74e6eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8404bf57902c4cd39efb499dea74e6eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34733 } } }
I20260927 12:50:25.620344 21507 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8404bf57902c4cd39efb499dea74e6eb [sys.catalog]: This master's current role is: LEADER
I20260927 12:50:25.620630 21515 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:50:25.621639 21498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:25.616710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.112:46841 (local address 127.0.0.1:34733)
0927 12:50:25.616834 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:25.616838 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:25.617001 (+ 163us) server_negotiation.cc:407] Connection header received
0927 12:50:25.617126 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:25.617129 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:25.617170 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:25.617231 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:25.617523 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:25.618303 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:25.619407 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:25.619729 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:25.620228 (+ 499us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:25.621013 (+ 785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:25.621017 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:25.621020 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:50:25.621033 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:25.621061 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:50:25.621073 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:25.621428 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:25.621483 (+ 55us) server_negotiation.cc:299] Negotiation successful
0927 12:50:25.621516 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 12:50:25.624415 21515 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:50:25.629894 21515 catalog_manager.cc:1347] Generated new cluster ID: ee0aa2f0360744449c5025b426ecb77c
I20260927 12:50:25.630000 21515 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:50:25.655514 21515 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:50:25.656544 21515 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:50:25.673622 21515 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8404bf57902c4cd39efb499dea74e6eb: Generated new TSK 0
I20260927 12:50:25.674415 21515 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:50:25.685868 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 21427
W20260927 12:50:25.690636 21412 connection.cc:535] client connection to 127.0.0.1:34733 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34733): Connection reset by peer (error 104)
2026-09-27T12:50:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1362 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790513161160093-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:50:25 dist-test-slave-lkn4 krb5kdc[21526](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:50:25 dist-test-slave-lkn4 krb5kdc[21526](info): set up 2 sockets
Sep 27 12:50:25 dist-test-slave-lkn4 krb5kdc[21526](info): commencing operation
krb5kdc: starting...
W20260927 12:50:26.646207 392 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790513161160093-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790513161160093-392-0/krb5kdc/test-user.keytab.
Sep 27 12:50:26 dist-test-slave-lkn4 krb5kdc[21526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513426, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:50:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:50:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790513161160093-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790513161160093-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 12:50:26.763098 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790513161160093-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790513161160093-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790513161160093-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35363
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790513161160093-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35363
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:50:26.874173 21539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:50:26.874457 21539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:50:26.874518 21539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:50:26.878579 21539 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:50:26.878638 21539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:50:26.878664 21539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:50:26.878686 21539 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:50:26.878714 21539 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:50:26.884033 21539 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790513161160093-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790513161160093-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35363
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790513161160093-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790513161160093-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:50:26.885057 21539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:50:26.886101 21539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:50:26.894255 21547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:50:26.894311 21544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:50:26.894359 21545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:50:26.894426 21539 server_base.cc:1034] running on GCE node
I20260927 12:50:26.895011 21539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:50:26.896095 21539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:50:26.897279 21539 hybrid_clock.cc:648] HybridClock initialized: now 1790513426897250 us; error 39 us; skew 500 ppm
Sep 27 12:50:26 dist-test-slave-lkn4 krb5kdc[21526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513426, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:50:26.900132 21539 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 12:50:26.901041 21539 webserver.cc:460] Webserver started at http://127.0.0.1:44477/ using document root <none> and password file <none>
I20260927 12:50:26.901569 21539 fs_manager.cc:359] Metadata directory not provided
I20260927 12:50:26.901650 21539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:50:26.901851 21539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:50:26.903904 21539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790513161160093-392-0/minicluster-data/master-0/data/instance:
uuid: "8a7b42e1de5346ee861e2a56610d1570"
format_stamp: "Formatted at 2026-09-27 12:50:26 on dist-test-slave-lkn4"
server_key: "bcac21fe17a8795af2ebddd52703e59a"
server_key_iv: "68633aab7b0493fa9562291645006d39"
server_key_version: "encryptionkey@0"
I20260927 12:50:26.904455 21539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790513161160093-392-0/minicluster-data/master-0/wal/instance:
uuid: "8a7b42e1de5346ee861e2a56610d1570"
format_stamp: "Formatted at 2026-09-27 12:50:26 on dist-test-slave-lkn4"
server_key: "bcac21fe17a8795af2ebddd52703e59a"
server_key_iv: "68633aab7b0493fa9562291645006d39"
server_key_version: "encryptionkey@0"
I20260927 12:50:26.908866 21539 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260927 12:50:26.911396 21554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:26.912753 21539 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 12:50:26.912879 21539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790513161160093-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790513161160093-392-0/minicluster-data/master-0/wal
uuid: "8a7b42e1de5346ee861e2a56610d1570"
format_stamp: "Formatted at 2026-09-27 12:50:26 on dist-test-slave-lkn4"
server_key: "bcac21fe17a8795af2ebddd52703e59a"
server_key_iv: "68633aab7b0493fa9562291645006d39"
server_key_version: "encryptionkey@0"
I20260927 12:50:26.912984 21539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790513161160093-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790513161160093-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790513161160093-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:50:26.917773 21539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:50:26.920964 21539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:50:26.921139 21539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:50:26.928164 21539 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35363
I20260927 12:50:26.928234 21606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35363 every 8 connection(s)
I20260927 12:50:26.929165 21539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:50:26.932126 21607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:50:26.937799 21607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a7b42e1de5346ee861e2a56610d1570: Bootstrap starting.
I20260927 12:50:26.939126 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 21539
I20260927 12:50:26.939270 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790513161160093-392-0/minicluster-data/master-0/wal/instance
I20260927 12:50:26.939523 392 external_mini_cluster.cc:1428] Setting key 96860bd43d825370d8c1f7ff0d29cfb0
I20260927 12:50:26.940508 21607 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a7b42e1de5346ee861e2a56610d1570: Neither blocks nor log segments found. Creating new log.
I20260927 12:50:26.941501 21607 log.cc:824] T 00000000000000000000000000000000 P 8a7b42e1de5346ee861e2a56610d1570: Log is configured to *not* fsync() on all Append() calls
I20260927 12:50:26.943825 21607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a7b42e1de5346ee861e2a56610d1570: No bootstrap required, opened a new log
I20260927 12:50:26.944998 21610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:26.940687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.112:54997 (local address 127.0.0.1:35363)
0927 12:50:26.941150 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:26.941168 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:26.941211 (+ 43us) server_negotiation.cc:407] Connection header received
0927 12:50:26.942070 (+ 859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:26.942096 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:26.942433 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:26.942803 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:26.943491 (+ 688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:26.943497 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:26.943511 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:50:26.943523 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:26.943588 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:50:26.943614 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:26.943723 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:26.943886 (+ 163us) server_negotiation.cc:299] Negotiation successful
0927 12:50:26.944048 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":136,"threads_started":1}
I20260927 12:50:26.948047 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:50:26.948302 21607 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a7b42e1de5346ee861e2a56610d1570 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a7b42e1de5346ee861e2a56610d1570" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35363 } }
I20260927 12:50:26.948534 21607 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a7b42e1de5346ee861e2a56610d1570 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:50:26.948582 21607 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a7b42e1de5346ee861e2a56610d1570 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a7b42e1de5346ee861e2a56610d1570, State: Initialized, Role: FOLLOWER
I20260927 12:50:26.948995 21607 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a7b42e1de5346ee861e2a56610d1570 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a7b42e1de5346ee861e2a56610d1570" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35363 } }
I20260927 12:50:26.949122 21607 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a7b42e1de5346ee861e2a56610d1570 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:50:26.949187 21607 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a7b42e1de5346ee861e2a56610d1570 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:50:26.949277 21607 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a7b42e1de5346ee861e2a56610d1570 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:50:26.950518 21607 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a7b42e1de5346ee861e2a56610d1570 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a7b42e1de5346ee861e2a56610d1570" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35363 } }
I20260927 12:50:26.950861 21607 leader_election.cc:302] T 00000000000000000000000000000000 P 8a7b42e1de5346ee861e2a56610d1570 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a7b42e1de5346ee861e2a56610d1570; no voters:
I20260927 12:50:26.951192 21607 leader_election.cc:288] T 00000000000000000000000000000000 P 8a7b42e1de5346ee861e2a56610d1570 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:50:26.951299 21617 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a7b42e1de5346ee861e2a56610d1570 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:50:26.951519 21617 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a7b42e1de5346ee861e2a56610d1570 [term 1 LEADER]: Becoming Leader. State: Replica: 8a7b42e1de5346ee861e2a56610d1570, State: Running, Role: LEADER
I20260927 12:50:26.952103 21610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:26.950236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.112:40605 (local address 127.0.0.1:35363)
0927 12:50:26.950398 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:26.950402 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:26.950498 (+ 96us) server_negotiation.cc:407] Connection header received
0927 12:50:26.950630 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:26.950633 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:26.950670 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:26.950743 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:26.951585 (+ 842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:26.951589 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:26.951591 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:50:26.951605 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:26.951637 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:50:26.951652 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:26.951728 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:26.951905 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 12:50:26.951949 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260927 12:50:26.951980 21617 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a7b42e1de5346ee861e2a56610d1570 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a7b42e1de5346ee861e2a56610d1570" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35363 } }
I20260927 12:50:26.952628 21607 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a7b42e1de5346ee861e2a56610d1570 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:50:26.953874 21619 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a7b42e1de5346ee861e2a56610d1570 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a7b42e1de5346ee861e2a56610d1570. Latest consensus state: current_term: 1 leader_uuid: "8a7b42e1de5346ee861e2a56610d1570" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a7b42e1de5346ee861e2a56610d1570" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35363 } } }
I20260927 12:50:26.953982 21619 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a7b42e1de5346ee861e2a56610d1570 [sys.catalog]: This master's current role is: LEADER
I20260927 12:50:26.954284 21618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a7b42e1de5346ee861e2a56610d1570 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a7b42e1de5346ee861e2a56610d1570" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a7b42e1de5346ee861e2a56610d1570" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35363 } } }
I20260927 12:50:26.954372 21618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a7b42e1de5346ee861e2a56610d1570 [sys.catalog]: This master's current role is: LEADER
I20260927 12:50:26.954466 21626 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:50:26.957661 21626 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:50:26.963819 21626 catalog_manager.cc:1347] Generated new cluster ID: 4638b29e42df403fb44c92c64d32f52b
I20260927 12:50:26.963917 21626 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:50:26.979584 21626 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:50:26.979938 21626 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:50:26.993134 21626 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a7b42e1de5346ee861e2a56610d1570: Generated new TSK 0
I20260927 12:50:26.994102 21626 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:50:27.000877 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 21539
2026-09-27T12:50:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1315 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790513161160093-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:50:27 dist-test-slave-lkn4 krb5kdc[21638](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:50:27 dist-test-slave-lkn4 krb5kdc[21638](info): set up 2 sockets
Sep 27 12:50:27 dist-test-slave-lkn4 krb5kdc[21638](info): commencing operation
krb5kdc: starting...
W20260927 12:50:27.962164 392 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790513161160093-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790513161160093-392-0/krb5kdc/test-user.keytab.
Sep 27 12:50:28 dist-test-slave-lkn4 krb5kdc[21638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:50:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:50:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790513161160093-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790513161160093-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 12:50:28.083412 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790513161160093-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790513161160093-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790513161160093-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43817
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790513161160093-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43817
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:50:28.193184 21651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:50:28.193485 21651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:50:28.193547 21651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:50:28.197790 21651 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:50:28.197881 21651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:50:28.197909 21651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:50:28.197930 21651 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:50:28.197952 21651 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:50:28.203467 21651 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790513161160093-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790513161160093-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43817
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790513161160093-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790513161160093-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:50:28.204612 21651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:50:28.205663 21651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:50:28.214129 21656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:50:28.214129 21657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:50:28.214133 21659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:50:28.214499 21651 server_base.cc:1034] running on GCE node
I20260927 12:50:28.215059 21651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:50:28.216158 21651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:50:28.217329 21651 hybrid_clock.cc:648] HybridClock initialized: now 1790513428217309 us; error 30 us; skew 500 ppm
Sep 27 12:50:28 dist-test-slave-lkn4 krb5kdc[21638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513428, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:50:28.220126 21651 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 12:50:28.221014 21651 webserver.cc:460] Webserver started at http://127.0.0.1:37715/ using document root <none> and password file <none>
I20260927 12:50:28.221541 21651 fs_manager.cc:359] Metadata directory not provided
I20260927 12:50:28.221604 21651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:50:28.221853 21651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:50:28.223914 21651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790513161160093-392-0/minicluster-data/master-0/data/instance:
uuid: "a9293e13206448238673072b7ba94e7b"
format_stamp: "Formatted at 2026-09-27 12:50:28 on dist-test-slave-lkn4"
server_key: "508d9faf449119173c5927ee6afda1fd"
server_key_iv: "aed5b5a5e4ab773c41f984bbc970ef56"
server_key_version: "encryptionkey@0"
I20260927 12:50:28.224431 21651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790513161160093-392-0/minicluster-data/master-0/wal/instance:
uuid: "a9293e13206448238673072b7ba94e7b"
format_stamp: "Formatted at 2026-09-27 12:50:28 on dist-test-slave-lkn4"
server_key: "508d9faf449119173c5927ee6afda1fd"
server_key_iv: "aed5b5a5e4ab773c41f984bbc970ef56"
server_key_version: "encryptionkey@0"
I20260927 12:50:28.228127 21651 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 12:50:28.230558 21666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:28.231972 21651 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:50:28.232111 21651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790513161160093-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790513161160093-392-0/minicluster-data/master-0/wal
uuid: "a9293e13206448238673072b7ba94e7b"
format_stamp: "Formatted at 2026-09-27 12:50:28 on dist-test-slave-lkn4"
server_key: "508d9faf449119173c5927ee6afda1fd"
server_key_iv: "aed5b5a5e4ab773c41f984bbc970ef56"
server_key_version: "encryptionkey@0"
I20260927 12:50:28.232218 21651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790513161160093-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790513161160093-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790513161160093-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:50:28.265100 21651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:50:28.268442 21651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:50:28.268643 21651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:50:28.275547 21651 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43817
I20260927 12:50:28.275558 21718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43817 every 8 connection(s)
I20260927 12:50:28.276513 21651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:50:28.279446 21719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:50:28.280179 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 21651
I20260927 12:50:28.280315 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790513161160093-392-0/minicluster-data/master-0/wal/instance
I20260927 12:50:28.280613 392 external_mini_cluster.cc:1428] Setting key 7aa7b5856ebb333d16730dc440d78bd7
I20260927 12:50:28.285672 21719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9293e13206448238673072b7ba94e7b: Bootstrap starting.
I20260927 12:50:28.288388 21719 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9293e13206448238673072b7ba94e7b: Neither blocks nor log segments found. Creating new log.
I20260927 12:50:28.289259 21719 log.cc:824] T 00000000000000000000000000000000 P a9293e13206448238673072b7ba94e7b: Log is configured to *not* fsync() on all Append() calls
Sep 27 12:50:28 dist-test-slave-lkn4 krb5kdc[21638](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 12:50:28.291157 21719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9293e13206448238673072b7ba94e7b: No bootstrap required, opened a new log
I20260927 12:50:28.294428 21722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:28.281935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.112:40061 (local address 127.0.0.1:43817)
0927 12:50:28.282368 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:28.282382 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:28.282417 (+ 35us) server_negotiation.cc:407] Connection header received
0927 12:50:28.283051 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:28.283068 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:28.283419 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:28.283690 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:28.284546 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:28.285568 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:28.286704 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:28.287174 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:28.287292 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:28.289992 (+ 2700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:28.289997 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:28.290011 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:50:28.290057 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:28.291667 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:28.292131 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:28.292136 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:28.292141 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:28.292248 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:28.292583 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:28.292586 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:28.292587 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:28.292970 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:28.293153 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:28.293458 (+ 305us) server_negotiation.cc:299] Negotiation successful
0927 12:50:28.293662 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":115,"threads_started":1}
I20260927 12:50:28.294871 21719 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9293e13206448238673072b7ba94e7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9293e13206448238673072b7ba94e7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43817 } }
I20260927 12:50:28.295099 21719 raft_consensus.cc:374] T 00000000000000000000000000000000 P a9293e13206448238673072b7ba94e7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:50:28.295163 21719 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9293e13206448238673072b7ba94e7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9293e13206448238673072b7ba94e7b, State: Initialized, Role: FOLLOWER
I20260927 12:50:28.295598 21719 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9293e13206448238673072b7ba94e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9293e13206448238673072b7ba94e7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43817 } }
I20260927 12:50:28.295763 21719 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9293e13206448238673072b7ba94e7b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:50:28.295825 21719 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9293e13206448238673072b7ba94e7b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:50:28.295914 21719 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9293e13206448238673072b7ba94e7b [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:50:28.296963 21719 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9293e13206448238673072b7ba94e7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9293e13206448238673072b7ba94e7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43817 } }
I20260927 12:50:28.297326 21719 leader_election.cc:302] T 00000000000000000000000000000000 P a9293e13206448238673072b7ba94e7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9293e13206448238673072b7ba94e7b; no voters:
I20260927 12:50:28.297515 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:50:28.297709 21719 leader_election.cc:288] T 00000000000000000000000000000000 P a9293e13206448238673072b7ba94e7b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:50:28.297731 21724 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9293e13206448238673072b7ba94e7b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:50:28.297911 21724 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9293e13206448238673072b7ba94e7b [term 1 LEADER]: Becoming Leader. State: Replica: a9293e13206448238673072b7ba94e7b, State: Running, Role: LEADER
I20260927 12:50:28.298449 21724 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9293e13206448238673072b7ba94e7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9293e13206448238673072b7ba94e7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43817 } }
I20260927 12:50:28.299261 21719 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9293e13206448238673072b7ba94e7b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:50:28.300096 21725 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9293e13206448238673072b7ba94e7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9293e13206448238673072b7ba94e7b. Latest consensus state: current_term: 1 leader_uuid: "a9293e13206448238673072b7ba94e7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9293e13206448238673072b7ba94e7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43817 } } }
I20260927 12:50:28.300235 21725 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9293e13206448238673072b7ba94e7b [sys.catalog]: This master's current role is: LEADER
I20260927 12:50:28.300542 21726 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9293e13206448238673072b7ba94e7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a9293e13206448238673072b7ba94e7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9293e13206448238673072b7ba94e7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43817 } } }
I20260927 12:50:28.300683 21726 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9293e13206448238673072b7ba94e7b [sys.catalog]: This master's current role is: LEADER
I20260927 12:50:28.301610 21736 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:50:28.304708 21736 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:50:28.309237 21722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:28.299495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.112:54055 (local address 127.0.0.1:43817)
0927 12:50:28.299597 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:28.299601 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:28.300013 (+ 412us) server_negotiation.cc:407] Connection header received
0927 12:50:28.300195 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:28.300198 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:28.300246 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:28.300309 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:28.301181 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:28.301912 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:28.303076 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:28.303443 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:28.303561 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:28.305216 (+ 1655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:28.305221 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:28.305224 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:50:28.305252 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:28.306984 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:28.307634 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:28.307638 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:28.307640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:28.307733 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:28.308210 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:28.308214 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:28.308215 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:28.308446 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:28.308579 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:28.308944 (+ 365us) server_negotiation.cc:299] Negotiation successful
0927 12:50:28.309056 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 12:50:28.310252 21736 catalog_manager.cc:1347] Generated new cluster ID: 3e3f9f8745e54c1f9675571288d42650
I20260927 12:50:28.310456 21736 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:50:28.319046 21736 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:50:28.320315 21736 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:50:28.326612 21736 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a9293e13206448238673072b7ba94e7b: Generated new TSK 0
I20260927 12:50:28.327443 21736 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:50:28.337816 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 21651
W20260927 12:50:28.342801 21636 connection.cc:535] client connection to 127.0.0.1:43817 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43817): Connection reset by peer (error 104)
2026-09-27T12:50:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1336 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790513161160093-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:50:28 dist-test-slave-lkn4 krb5kdc[21750](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:50:28 dist-test-slave-lkn4 krb5kdc[21750](info): set up 2 sockets
Sep 27 12:50:28 dist-test-slave-lkn4 krb5kdc[21750](info): commencing operation
krb5kdc: starting...
W20260927 12:50:29.301990 392 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790513161160093-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790513161160093-392-0/krb5kdc/test-user.keytab.
Sep 27 12:50:29 dist-test-slave-lkn4 krb5kdc[21750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:50:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:50:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790513161160093-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790513161160093-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 12:50:29.425647 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790513161160093-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790513161160093-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790513161160093-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41711
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790513161160093-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41711
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:50:29.536533 21763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:50:29.536847 21763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:50:29.536908 21763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:50:29.541111 21763 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:50:29.541200 21763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:50:29.541229 21763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:50:29.541250 21763 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:50:29.541280 21763 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:50:29.546832 21763 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790513161160093-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790513161160093-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41711
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790513161160093-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790513161160093-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:50:29.547916 21763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:50:29.548908 21763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 12:50:29.557531 21763 server_base.cc:1034] running on GCE node
W20260927 12:50:29.557454 21769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:50:29.557457 21768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:50:29.557463 21771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:50:29.558452 21763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:50:29.559660 21763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:50:29.560843 21763 hybrid_clock.cc:648] HybridClock initialized: now 1790513429560820 us; error 34 us; skew 500 ppm
Sep 27 12:50:29 dist-test-slave-lkn4 krb5kdc[21750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513429, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:50:29.563649 21763 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 12:50:29.564611 21763 webserver.cc:460] Webserver started at http://127.0.0.1:34059/ using document root <none> and password file <none>
I20260927 12:50:29.565158 21763 fs_manager.cc:359] Metadata directory not provided
I20260927 12:50:29.565227 21763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:50:29.565452 21763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:50:29.567664 21763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790513161160093-392-0/minicluster-data/master-0/data/instance:
uuid: "10be96c8995248f9b371c541cb5129f3"
format_stamp: "Formatted at 2026-09-27 12:50:29 on dist-test-slave-lkn4"
server_key: "89aad27bf747d3bf184638416ba8bbed"
server_key_iv: "e34bc992bf56d82a86d88886adb49760"
server_key_version: "encryptionkey@0"
I20260927 12:50:29.568209 21763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790513161160093-392-0/minicluster-data/master-0/wal/instance:
uuid: "10be96c8995248f9b371c541cb5129f3"
format_stamp: "Formatted at 2026-09-27 12:50:29 on dist-test-slave-lkn4"
server_key: "89aad27bf747d3bf184638416ba8bbed"
server_key_iv: "e34bc992bf56d82a86d88886adb49760"
server_key_version: "encryptionkey@0"
I20260927 12:50:29.572243 21763 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260927 12:50:29.574990 21778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:29.576392 21763 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 12:50:29.576507 21763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790513161160093-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790513161160093-392-0/minicluster-data/master-0/wal
uuid: "10be96c8995248f9b371c541cb5129f3"
format_stamp: "Formatted at 2026-09-27 12:50:29 on dist-test-slave-lkn4"
server_key: "89aad27bf747d3bf184638416ba8bbed"
server_key_iv: "e34bc992bf56d82a86d88886adb49760"
server_key_version: "encryptionkey@0"
I20260927 12:50:29.576637 21763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790513161160093-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790513161160093-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790513161160093-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:50:29.599834 21763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:50:29.603226 21763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:50:29.603410 21763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:50:29.610260 21763 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41711
I20260927 12:50:29.610288 21830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41711 every 8 connection(s)
I20260927 12:50:29.611279 21763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:50:29.612313 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 21763
I20260927 12:50:29.612447 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790513161160093-392-0/minicluster-data/master-0/wal/instance
I20260927 12:50:29.612732 392 external_mini_cluster.cc:1428] Setting key a380f851dd6df995326c126b418291c7
I20260927 12:50:29.614728 21831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:50:29.620371 21831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10be96c8995248f9b371c541cb5129f3: Bootstrap starting.
I20260927 12:50:29.621937 21834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:29.614256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.112:42145 (local address 127.0.0.1:41711)
0927 12:50:29.614775 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:29.614792 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:29.614845 (+ 53us) server_negotiation.cc:407] Connection header received
0927 12:50:29.615521 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:29.615539 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:29.615908 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:29.616241 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:29.616659 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:29.617668 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:29.618604 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:29.619080 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:29.619224 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:29.620179 (+ 955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:29.620184 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:29.620199 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:50:29.620607 (+ 408us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:29.620663 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:50:29.620686 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:29.620805 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:29.621003 (+ 198us) server_negotiation.cc:299] Negotiation successful
0927 12:50:29.621139 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":132,"threads_started":1}
I20260927 12:50:29.622700 21831 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10be96c8995248f9b371c541cb5129f3: Neither blocks nor log segments found. Creating new log.
I20260927 12:50:29.623580 21831 log.cc:824] T 00000000000000000000000000000000 P 10be96c8995248f9b371c541cb5129f3: Log is configured to *not* fsync() on all Append() calls
I20260927 12:50:29.625104 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:50:29.625876 21831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10be96c8995248f9b371c541cb5129f3: No bootstrap required, opened a new log
I20260927 12:50:29.629772 21831 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10be96c8995248f9b371c541cb5129f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10be96c8995248f9b371c541cb5129f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41711 } }
I20260927 12:50:29.630062 21831 raft_consensus.cc:374] T 00000000000000000000000000000000 P 10be96c8995248f9b371c541cb5129f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:50:29.630126 21831 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10be96c8995248f9b371c541cb5129f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10be96c8995248f9b371c541cb5129f3, State: Initialized, Role: FOLLOWER
I20260927 12:50:29.630577 21831 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10be96c8995248f9b371c541cb5129f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10be96c8995248f9b371c541cb5129f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41711 } }
I20260927 12:50:29.630726 21831 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10be96c8995248f9b371c541cb5129f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:50:29.630793 21831 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10be96c8995248f9b371c541cb5129f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:50:29.630909 21831 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10be96c8995248f9b371c541cb5129f3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:50:29.631312 21834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:29.627371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.112:34235 (local address 127.0.0.1:41711)
0927 12:50:29.627500 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:29.627504 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:29.627679 (+ 175us) server_negotiation.cc:407] Connection header received
0927 12:50:29.627853 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:29.627856 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:29.627898 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:29.627959 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:29.628220 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:29.628746 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:29.629550 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:29.629794 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:29.629879 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:29.630694 (+ 815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:29.630697 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:29.630700 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:50:29.630711 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:29.630740 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:50:29.630752 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:29.630836 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:29.631024 (+ 188us) server_negotiation.cc:299] Negotiation successful
0927 12:50:29.631200 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 12:50:29.632028 21831 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10be96c8995248f9b371c541cb5129f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10be96c8995248f9b371c541cb5129f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41711 } }
I20260927 12:50:29.632392 21831 leader_election.cc:302] T 00000000000000000000000000000000 P 10be96c8995248f9b371c541cb5129f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10be96c8995248f9b371c541cb5129f3; no voters:
I20260927 12:50:29.632691 21831 leader_election.cc:288] T 00000000000000000000000000000000 P 10be96c8995248f9b371c541cb5129f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:50:29.632795 21841 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10be96c8995248f9b371c541cb5129f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:50:29.632990 21841 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10be96c8995248f9b371c541cb5129f3 [term 1 LEADER]: Becoming Leader. State: Replica: 10be96c8995248f9b371c541cb5129f3, State: Running, Role: LEADER
I20260927 12:50:29.633329 21841 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10be96c8995248f9b371c541cb5129f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10be96c8995248f9b371c541cb5129f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41711 } }
I20260927 12:50:29.633917 21831 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10be96c8995248f9b371c541cb5129f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:50:29.634964 21843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10be96c8995248f9b371c541cb5129f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10be96c8995248f9b371c541cb5129f3. Latest consensus state: current_term: 1 leader_uuid: "10be96c8995248f9b371c541cb5129f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10be96c8995248f9b371c541cb5129f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41711 } } }
I20260927 12:50:29.635095 21843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10be96c8995248f9b371c541cb5129f3 [sys.catalog]: This master's current role is: LEADER
I20260927 12:50:29.635476 21850 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:50:29.635447 21842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10be96c8995248f9b371c541cb5129f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "10be96c8995248f9b371c541cb5129f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10be96c8995248f9b371c541cb5129f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41711 } } }
I20260927 12:50:29.635597 21842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10be96c8995248f9b371c541cb5129f3 [sys.catalog]: This master's current role is: LEADER
I20260927 12:50:29.639199 21850 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:50:29.644614 21850 catalog_manager.cc:1347] Generated new cluster ID: 45bc541a78854f6abeaa882edb63e311
I20260927 12:50:29.644704 21850 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:50:29.672799 21850 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:50:29.673789 21850 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:50:29.680781 21850 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 10be96c8995248f9b371c541cb5129f3: Generated new TSK 0
I20260927 12:50:29.681547 21850 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:50:29.686925 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 21763
W20260927 12:50:29.691749 21748 connection.cc:535] client connection to 127.0.0.1:41711 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41711): Connection reset by peer (error 104)
2026-09-27T12:50:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1348 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790513161160093-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:50:29 dist-test-slave-lkn4 krb5kdc[21862](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:50:29 dist-test-slave-lkn4 krb5kdc[21862](info): set up 2 sockets
Sep 27 12:50:29 dist-test-slave-lkn4 krb5kdc[21862](info): commencing operation
krb5kdc: starting...
W20260927 12:50:30.642120 392 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790513161160093-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790513161160093-392-0/krb5kdc/test-user.keytab.
Sep 27 12:50:30 dist-test-slave-lkn4 krb5kdc[21862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:50:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:50:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790513161160093-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790513161160093-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 12:50:30.770001 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790513161160093-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790513161160093-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790513161160093-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37211
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790513161160093-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37211
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:50:30.880672 21875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:50:30.880950 21875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:50:30.881009 21875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:50:30.885010 21875 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:50:30.885078 21875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:50:30.885104 21875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:50:30.885126 21875 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:50:30.885146 21875 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:50:30.890614 21875 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790513161160093-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790513161160093-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37211
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790513161160093-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790513161160093-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:50:30.891888 21875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:50:30.892975 21875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:50:30.901588 21880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:50:30.901651 21883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:50:30.901669 21875 server_base.cc:1034] running on GCE node
W20260927 12:50:30.901588 21881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:50:30.902514 21875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:50:30.903627 21875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:50:30.904878 21875 hybrid_clock.cc:648] HybridClock initialized: now 1790513430904799 us; error 29 us; skew 500 ppm
Sep 27 12:50:30 dist-test-slave-lkn4 krb5kdc[21862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513430, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:50:30.911641 21875 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 12:50:30.912703 21875 webserver.cc:460] Webserver started at http://127.0.0.1:43983/ using document root <none> and password file <none>
I20260927 12:50:30.913293 21875 fs_manager.cc:359] Metadata directory not provided
I20260927 12:50:30.913360 21875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:50:30.913559 21875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:50:30.915755 21875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790513161160093-392-0/minicluster-data/master-0/data/instance:
uuid: "201cf650d3dc4e77948a875a17f700b0"
format_stamp: "Formatted at 2026-09-27 12:50:30 on dist-test-slave-lkn4"
server_key: "a371f596cde6b4f9a306e146868da747"
server_key_iv: "0b2594a74035dd0ccb576853a5f28d66"
server_key_version: "encryptionkey@0"
I20260927 12:50:30.916301 21875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790513161160093-392-0/minicluster-data/master-0/wal/instance:
uuid: "201cf650d3dc4e77948a875a17f700b0"
format_stamp: "Formatted at 2026-09-27 12:50:30 on dist-test-slave-lkn4"
server_key: "a371f596cde6b4f9a306e146868da747"
server_key_iv: "0b2594a74035dd0ccb576853a5f28d66"
server_key_version: "encryptionkey@0"
I20260927 12:50:30.920253 21875 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 12:50:30.922933 21890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:30.924398 21875 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:50:30.924540 21875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790513161160093-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790513161160093-392-0/minicluster-data/master-0/wal
uuid: "201cf650d3dc4e77948a875a17f700b0"
format_stamp: "Formatted at 2026-09-27 12:50:30 on dist-test-slave-lkn4"
server_key: "a371f596cde6b4f9a306e146868da747"
server_key_iv: "0b2594a74035dd0ccb576853a5f28d66"
server_key_version: "encryptionkey@0"
I20260927 12:50:30.924656 21875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790513161160093-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790513161160093-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790513161160093-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:50:30.929572 21875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:50:30.932842 21875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:50:30.933034 21875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:50:30.940500 21875 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37211
I20260927 12:50:30.940552 21942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37211 every 8 connection(s)
I20260927 12:50:30.941504 21875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:50:30.944538 21943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:50:30.946329 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 21875
I20260927 12:50:30.946458 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790513161160093-392-0/minicluster-data/master-0/wal/instance
I20260927 12:50:30.946719 392 external_mini_cluster.cc:1428] Setting key 895bdfbce7cc9ed3892ccb6caca78d6d
I20260927 12:50:30.951138 21943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 201cf650d3dc4e77948a875a17f700b0: Bootstrap starting.
I20260927 12:50:30.952056 21946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:30.948143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.112:33363 (local address 127.0.0.1:37211)
0927 12:50:30.948611 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:30.948632 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:30.948670 (+ 38us) server_negotiation.cc:407] Connection header received
0927 12:50:30.949323 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:30.949349 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:30.949652 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:30.949980 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:30.950829 (+ 849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:30.950835 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:30.950849 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:50:30.950859 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:30.950914 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:50:30.950934 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:30.951065 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:30.951134 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 12:50:30.951296 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":138,"threads_started":1}
I20260927 12:50:30.953673 21943 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 201cf650d3dc4e77948a875a17f700b0: Neither blocks nor log segments found. Creating new log.
I20260927 12:50:30.954520 21943 log.cc:824] T 00000000000000000000000000000000 P 201cf650d3dc4e77948a875a17f700b0: Log is configured to *not* fsync() on all Append() calls
I20260927 12:50:30.954947 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:50:30.956485 21943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 201cf650d3dc4e77948a875a17f700b0: No bootstrap required, opened a new log
I20260927 12:50:30.958639 21946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:30.956879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.112:32943 (local address 127.0.0.1:37211)
0927 12:50:30.956983 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:30.956988 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:30.957179 (+ 191us) server_negotiation.cc:407] Connection header received
0927 12:50:30.957303 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:30.957306 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:30.957340 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:30.957411 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:30.958239 (+ 828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:30.958242 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:30.958245 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:50:30.958256 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:30.958283 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:50:30.958293 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:30.958360 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:30.958476 (+ 116us) server_negotiation.cc:299] Negotiation successful
0927 12:50:30.958516 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 12:50:30.960454 21943 raft_consensus.cc:348] T 00000000000000000000000000000000 P 201cf650d3dc4e77948a875a17f700b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "201cf650d3dc4e77948a875a17f700b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37211 } }
I20260927 12:50:30.960683 21943 raft_consensus.cc:374] T 00000000000000000000000000000000 P 201cf650d3dc4e77948a875a17f700b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:50:30.960731 21943 raft_consensus.cc:729] T 00000000000000000000000000000000 P 201cf650d3dc4e77948a875a17f700b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 201cf650d3dc4e77948a875a17f700b0, State: Initialized, Role: FOLLOWER
I20260927 12:50:30.961144 21943 consensus_queue.cc:260] T 00000000000000000000000000000000 P 201cf650d3dc4e77948a875a17f700b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "201cf650d3dc4e77948a875a17f700b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37211 } }
I20260927 12:50:30.961264 21943 raft_consensus.cc:388] T 00000000000000000000000000000000 P 201cf650d3dc4e77948a875a17f700b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:50:30.961323 21943 raft_consensus.cc:482] T 00000000000000000000000000000000 P 201cf650d3dc4e77948a875a17f700b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:50:30.961413 21943 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 201cf650d3dc4e77948a875a17f700b0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:50:30.962383 21943 raft_consensus.cc:504] T 00000000000000000000000000000000 P 201cf650d3dc4e77948a875a17f700b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "201cf650d3dc4e77948a875a17f700b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37211 } }
I20260927 12:50:30.962700 21943 leader_election.cc:302] T 00000000000000000000000000000000 P 201cf650d3dc4e77948a875a17f700b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 201cf650d3dc4e77948a875a17f700b0; no voters:
I20260927 12:50:30.963019 21943 leader_election.cc:288] T 00000000000000000000000000000000 P 201cf650d3dc4e77948a875a17f700b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:50:30.963107 21953 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 201cf650d3dc4e77948a875a17f700b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:50:30.963328 21953 raft_consensus.cc:686] T 00000000000000000000000000000000 P 201cf650d3dc4e77948a875a17f700b0 [term 1 LEADER]: Becoming Leader. State: Replica: 201cf650d3dc4e77948a875a17f700b0, State: Running, Role: LEADER
I20260927 12:50:30.963732 21953 consensus_queue.cc:237] T 00000000000000000000000000000000 P 201cf650d3dc4e77948a875a17f700b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "201cf650d3dc4e77948a875a17f700b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37211 } }
I20260927 12:50:30.964283 21943 sys_catalog.cc:564] T 00000000000000000000000000000000 P 201cf650d3dc4e77948a875a17f700b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:50:30.965263 21955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 201cf650d3dc4e77948a875a17f700b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 201cf650d3dc4e77948a875a17f700b0. Latest consensus state: current_term: 1 leader_uuid: "201cf650d3dc4e77948a875a17f700b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "201cf650d3dc4e77948a875a17f700b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37211 } } }
I20260927 12:50:30.965386 21955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 201cf650d3dc4e77948a875a17f700b0 [sys.catalog]: This master's current role is: LEADER
I20260927 12:50:30.965693 21954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 201cf650d3dc4e77948a875a17f700b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "201cf650d3dc4e77948a875a17f700b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "201cf650d3dc4e77948a875a17f700b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37211 } } }
I20260927 12:50:30.965758 21962 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:50:30.966503 21954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 201cf650d3dc4e77948a875a17f700b0 [sys.catalog]: This master's current role is: LEADER
I20260927 12:50:30.969115 21962 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:50:30.975477 21962 catalog_manager.cc:1347] Generated new cluster ID: f115ecc2764c42ecb3e8742b6eb602e7
I20260927 12:50:30.975564 21962 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:50:30.987177 21962 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:50:30.987577 21962 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:50:31.007613 21962 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 201cf650d3dc4e77948a875a17f700b0: Generated new TSK 0
I20260927 12:50:31.008544 21962 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:50:31.014564 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 21875
2026-09-27T12:50:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1327 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790513161160093-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:50:31 dist-test-slave-lkn4 krb5kdc[21974](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:50:31 dist-test-slave-lkn4 krb5kdc[21974](info): set up 2 sockets
Sep 27 12:50:31 dist-test-slave-lkn4 krb5kdc[21974](info): commencing operation
krb5kdc: starting...
W20260927 12:50:31.974162 392 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790513161160093-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790513161160093-392-0/krb5kdc/test-user.keytab.
Sep 27 12:50:32 dist-test-slave-lkn4 krb5kdc[21974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:50:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:50:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:50:32.091236 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790513161160093-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790513161160093-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790513161160093-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34537
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34537
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:50:32.202250 21987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:50:32.202543 21987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:50:32.202586 21987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:50:32.207032 21987 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:50:32.207111 21987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:50:32.207131 21987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:50:32.207146 21987 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:50:32.207160 21987 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:50:32.212739 21987 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790513161160093-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790513161160093-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34537
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790513161160093-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:50:32.213862 21987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:50:32.214954 21987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:50:32.223541 21993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:50:32.223627 21987 server_base.cc:1034] running on GCE node
W20260927 12:50:32.223536 21992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:50:32.223536 21995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:50:32.224365 21987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:50:32.225472 21987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:50:32.226631 21987 hybrid_clock.cc:648] HybridClock initialized: now 1790513432226584 us; error 61 us; skew 500 ppm
Sep 27 12:50:32 dist-test-slave-lkn4 krb5kdc[21974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513432, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:50:32.229483 21987 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:50:32.230440 21987 webserver.cc:460] Webserver started at http://127.0.98.62:38723/ using document root <none> and password file <none>
I20260927 12:50:32.230984 21987 fs_manager.cc:359] Metadata directory not provided
I20260927 12:50:32.231050 21987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:50:32.231251 21987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:50:32.233361 21987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790513161160093-392-0/minicluster-data/master-0/data/instance:
uuid: "c7e21e3c4a2f413dbdff32799d0ea11a"
format_stamp: "Formatted at 2026-09-27 12:50:32 on dist-test-slave-lkn4"
server_key: "7232c69f3e0ff3e23703cf44f00682cb"
server_key_iv: "65442584adc8d70a3ddcfbbed2fff58f"
server_key_version: "encryptionkey@0"
I20260927 12:50:32.233904 21987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790513161160093-392-0/minicluster-data/master-0/wal/instance:
uuid: "c7e21e3c4a2f413dbdff32799d0ea11a"
format_stamp: "Formatted at 2026-09-27 12:50:32 on dist-test-slave-lkn4"
server_key: "7232c69f3e0ff3e23703cf44f00682cb"
server_key_iv: "65442584adc8d70a3ddcfbbed2fff58f"
server_key_version: "encryptionkey@0"
I20260927 12:50:32.237771 21987 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 12:50:32.240361 22002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:32.241613 21987 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:50:32.241736 21987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790513161160093-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790513161160093-392-0/minicluster-data/master-0/wal
uuid: "c7e21e3c4a2f413dbdff32799d0ea11a"
format_stamp: "Formatted at 2026-09-27 12:50:32 on dist-test-slave-lkn4"
server_key: "7232c69f3e0ff3e23703cf44f00682cb"
server_key_iv: "65442584adc8d70a3ddcfbbed2fff58f"
server_key_version: "encryptionkey@0"
I20260927 12:50:32.241851 21987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790513161160093-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790513161160093-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790513161160093-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:50:32.257892 21987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:50:32.261523 21987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:50:32.261811 21987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:50:32.269125 21987 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34537
I20260927 12:50:32.269173 22054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34537 every 8 connection(s)
I20260927 12:50:32.270219 21987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:50:32.273335 22055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:50:32.277649 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 21987
I20260927 12:50:32.277801 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790513161160093-392-0/minicluster-data/master-0/wal/instance
I20260927 12:50:32.278101 392 external_mini_cluster.cc:1428] Setting key 5818ecb51425d9c81d29e56eda2ca8e1
I20260927 12:50:32.279577 22055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7e21e3c4a2f413dbdff32799d0ea11a: Bootstrap starting.
I20260927 12:50:32.282212 22055 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7e21e3c4a2f413dbdff32799d0ea11a: Neither blocks nor log segments found. Creating new log.
I20260927 12:50:32.283037 22055 log.cc:824] T 00000000000000000000000000000000 P c7e21e3c4a2f413dbdff32799d0ea11a: Log is configured to *not* fsync() on all Append() calls
I20260927 12:50:32.285437 22055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7e21e3c4a2f413dbdff32799d0ea11a: No bootstrap required, opened a new log
Sep 27 12:50:32 dist-test-slave-lkn4 krb5kdc[21974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:50:32.289755 22055 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7e21e3c4a2f413dbdff32799d0ea11a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7e21e3c4a2f413dbdff32799d0ea11a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34537 } }
I20260927 12:50:32.290118 22055 raft_consensus.cc:374] T 00000000000000000000000000000000 P c7e21e3c4a2f413dbdff32799d0ea11a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:50:32.290187 22055 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7e21e3c4a2f413dbdff32799d0ea11a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7e21e3c4a2f413dbdff32799d0ea11a, State: Initialized, Role: FOLLOWER
I20260927 12:50:32.290771 22055 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7e21e3c4a2f413dbdff32799d0ea11a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7e21e3c4a2f413dbdff32799d0ea11a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34537 } }
I20260927 12:50:32.290922 22055 raft_consensus.cc:388] T 00000000000000000000000000000000 P c7e21e3c4a2f413dbdff32799d0ea11a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:50:32.290979 22055 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7e21e3c4a2f413dbdff32799d0ea11a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:50:32.291091 22055 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7e21e3c4a2f413dbdff32799d0ea11a [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:50:32.292390 22055 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7e21e3c4a2f413dbdff32799d0ea11a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7e21e3c4a2f413dbdff32799d0ea11a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34537 } }
I20260927 12:50:32.292834 22055 leader_election.cc:302] T 00000000000000000000000000000000 P c7e21e3c4a2f413dbdff32799d0ea11a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c7e21e3c4a2f413dbdff32799d0ea11a; no voters:
I20260927 12:50:32.293233 22055 leader_election.cc:288] T 00000000000000000000000000000000 P c7e21e3c4a2f413dbdff32799d0ea11a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:50:32.293421 22060 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7e21e3c4a2f413dbdff32799d0ea11a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:50:32.293645 22060 raft_consensus.cc:686] T 00000000000000000000000000000000 P c7e21e3c4a2f413dbdff32799d0ea11a [term 1 LEADER]: Becoming Leader. State: Replica: c7e21e3c4a2f413dbdff32799d0ea11a, State: Running, Role: LEADER
I20260927 12:50:32.294011 22060 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7e21e3c4a2f413dbdff32799d0ea11a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7e21e3c4a2f413dbdff32799d0ea11a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34537 } }
I20260927 12:50:32.294605 22058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:32.279496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56303 (local address 127.0.98.62:34537)
0927 12:50:32.279931 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:32.279947 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:32.279983 (+ 36us) server_negotiation.cc:407] Connection header received
0927 12:50:32.280682 (+ 699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:32.280701 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:32.281079 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:32.281375 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:32.282416 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:32.283534 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:32.284471 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:32.285226 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:32.285404 (+ 178us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:32.288129 (+ 2725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:32.288140 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:32.288164 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:50:32.288211 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:32.290497 (+ 2286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:32.291220 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:32.291228 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:32.291235 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:32.291402 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:32.291824 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:32.291828 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:32.291830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:32.292476 (+ 646us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:32.292784 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:32.293219 (+ 435us) server_negotiation.cc:299] Negotiation successful
0927 12:50:32.293504 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":138,"threads_started":1}
I20260927 12:50:32.294755 22055 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7e21e3c4a2f413dbdff32799d0ea11a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:50:32.295300 22060 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7e21e3c4a2f413dbdff32799d0ea11a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7e21e3c4a2f413dbdff32799d0ea11a. Latest consensus state: current_term: 1 leader_uuid: "c7e21e3c4a2f413dbdff32799d0ea11a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7e21e3c4a2f413dbdff32799d0ea11a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34537 } } }
I20260927 12:50:32.295591 22060 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7e21e3c4a2f413dbdff32799d0ea11a [sys.catalog]: This master's current role is: LEADER
I20260927 12:50:32.295955 22064 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:50:32.297343 22062 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7e21e3c4a2f413dbdff32799d0ea11a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c7e21e3c4a2f413dbdff32799d0ea11a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7e21e3c4a2f413dbdff32799d0ea11a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34537 } } }
I20260927 12:50:32.297452 22062 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7e21e3c4a2f413dbdff32799d0ea11a [sys.catalog]: This master's current role is: LEADER
I20260927 12:50:32.298161 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:50:32.299480 22064 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:50:32.306432 22064 catalog_manager.cc:1347] Generated new cluster ID: 10b872eb3aba4f728fe2f1d945d374a6
I20260927 12:50:32.306496 22064 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:50:32.309861 22058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:32.300876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40129 (local address 127.0.98.62:34537)
0927 12:50:32.301003 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:32.301009 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:32.301269 (+ 260us) server_negotiation.cc:407] Connection header received
0927 12:50:32.301408 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:32.301412 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:32.301461 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:32.301529 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:32.302442 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:32.303257 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:32.304378 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:32.304736 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:32.304838 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:32.306330 (+ 1492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:32.306334 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:32.306337 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:50:32.306366 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:32.308024 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:32.308526 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:32.308530 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:32.308531 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:32.308619 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:32.308947 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:32.308950 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:32.308952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:32.309138 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:32.309236 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:32.309597 (+ 361us) server_negotiation.cc:299] Negotiation successful
0927 12:50:32.309693 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 12:50:32.324470 22064 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:50:32.325477 22064 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:50:32.337766 22064 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c7e21e3c4a2f413dbdff32799d0ea11a: Generated new TSK 0
I20260927 12:50:32.338581 22064 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:50:32.343927 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 21987
W20260927 12:50:32.348538 21972 connection.cc:535] client connection to 127.0.98.62:34537 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:34537): Connection reset by peer (error 104)
2026-09-27T12:50:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1329 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790513161160093-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:50:32 dist-test-slave-lkn4 krb5kdc[22086](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:50:32 dist-test-slave-lkn4 krb5kdc[22086](info): set up 2 sockets
Sep 27 12:50:32 dist-test-slave-lkn4 krb5kdc[22086](info): commencing operation
krb5kdc: starting...
W20260927 12:50:33.310137 392 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790513161160093-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790513161160093-392-0/krb5kdc/test-user.keytab.
Sep 27 12:50:33 dist-test-slave-lkn4 krb5kdc[22086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:50:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:50:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:50:33.429332 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790513161160093-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790513161160093-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790513161160093-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38443
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38443
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:50:33.539023 22099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:50:33.539341 22099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:50:33.539403 22099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:50:33.543560 22099 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:50:33.543642 22099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:50:33.543668 22099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:50:33.543689 22099 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:50:33.543717 22099 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:50:33.549291 22099 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790513161160093-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790513161160093-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38443
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790513161160093-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:50:33.550400 22099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:50:33.551424 22099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 12:50:33.560084 22099 server_base.cc:1034] running on GCE node
W20260927 12:50:33.559998 22105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:50:33.559988 22104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:50:33.560047 22107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:50:33.560864 22099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:50:33.562085 22099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:50:33.563267 22099 hybrid_clock.cc:648] HybridClock initialized: now 1790513433563237 us; error 50 us; skew 500 ppm
Sep 27 12:50:33 dist-test-slave-lkn4 krb5kdc[22086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513433, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:50:33.566095 22099 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:50:33.567049 22099 webserver.cc:460] Webserver started at http://127.0.98.62:44649/ using document root <none> and password file <none>
I20260927 12:50:33.567597 22099 fs_manager.cc:359] Metadata directory not provided
I20260927 12:50:33.567667 22099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:50:33.567883 22099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:50:33.570024 22099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790513161160093-392-0/minicluster-data/master-0/data/instance:
uuid: "8a65c5b239e24b4492474bcae04bfe9e"
format_stamp: "Formatted at 2026-09-27 12:50:33 on dist-test-slave-lkn4"
server_key: "c4bde0b62e3c69c9c8cb07f3933d49f2"
server_key_iv: "e1ee7d69c6aae2f9b37e2b6d296d0431"
server_key_version: "encryptionkey@0"
I20260927 12:50:33.570611 22099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790513161160093-392-0/minicluster-data/master-0/wal/instance:
uuid: "8a65c5b239e24b4492474bcae04bfe9e"
format_stamp: "Formatted at 2026-09-27 12:50:33 on dist-test-slave-lkn4"
server_key: "c4bde0b62e3c69c9c8cb07f3933d49f2"
server_key_iv: "e1ee7d69c6aae2f9b37e2b6d296d0431"
server_key_version: "encryptionkey@0"
I20260927 12:50:33.574618 22099 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 12:50:33.577354 22114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:33.578768 22099 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:50:33.578903 22099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790513161160093-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790513161160093-392-0/minicluster-data/master-0/wal
uuid: "8a65c5b239e24b4492474bcae04bfe9e"
format_stamp: "Formatted at 2026-09-27 12:50:33 on dist-test-slave-lkn4"
server_key: "c4bde0b62e3c69c9c8cb07f3933d49f2"
server_key_iv: "e1ee7d69c6aae2f9b37e2b6d296d0431"
server_key_version: "encryptionkey@0"
I20260927 12:50:33.579015 22099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790513161160093-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790513161160093-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790513161160093-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:50:33.591945 22099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:50:33.595932 22099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:50:33.596138 22099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:50:33.603426 22099 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38443
I20260927 12:50:33.603502 22166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38443 every 8 connection(s)
I20260927 12:50:33.604445 22099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:50:33.605876 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 22099
I20260927 12:50:33.606017 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790513161160093-392-0/minicluster-data/master-0/wal/instance
I20260927 12:50:33.606344 392 external_mini_cluster.cc:1428] Setting key ee97ca9c041643e3e2e12dd9b91763d8
I20260927 12:50:33.607833 22167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:50:33.614722 22167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a65c5b239e24b4492474bcae04bfe9e: Bootstrap starting.
I20260927 12:50:33.616905 22170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:33.607810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38095 (local address 127.0.98.62:38443)
0927 12:50:33.608228 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:33.608247 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:33.608290 (+ 43us) server_negotiation.cc:407] Connection header received
0927 12:50:33.609127 (+ 837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:33.609150 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:33.609591 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:33.609971 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:33.610425 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:33.611913 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:33.612911 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:33.613622 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:33.613790 (+ 168us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:33.614819 (+ 1029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:33.614826 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:33.614843 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:50:33.615349 (+ 506us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:33.615424 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:50:33.615455 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:33.615579 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:33.615779 (+ 200us) server_negotiation.cc:299] Negotiation successful
0927 12:50:33.615925 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":115,"threads_started":1}
I20260927 12:50:33.617517 22167 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a65c5b239e24b4492474bcae04bfe9e: Neither blocks nor log segments found. Creating new log.
I20260927 12:50:33.618407 22167 log.cc:824] T 00000000000000000000000000000000 P 8a65c5b239e24b4492474bcae04bfe9e: Log is configured to *not* fsync() on all Append() calls
I20260927 12:50:33.620256 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:50:33.620504 22167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a65c5b239e24b4492474bcae04bfe9e: No bootstrap required, opened a new log
I20260927 12:50:33.624979 22167 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a65c5b239e24b4492474bcae04bfe9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a65c5b239e24b4492474bcae04bfe9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38443 } }
I20260927 12:50:33.625259 22167 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a65c5b239e24b4492474bcae04bfe9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:50:33.625319 22167 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a65c5b239e24b4492474bcae04bfe9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a65c5b239e24b4492474bcae04bfe9e, State: Initialized, Role: FOLLOWER
I20260927 12:50:33.625774 22167 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a65c5b239e24b4492474bcae04bfe9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a65c5b239e24b4492474bcae04bfe9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38443 } }
I20260927 12:50:33.625897 22167 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a65c5b239e24b4492474bcae04bfe9e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:50:33.625960 22167 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a65c5b239e24b4492474bcae04bfe9e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:50:33.626087 22167 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a65c5b239e24b4492474bcae04bfe9e [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:50:33.626720 22170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:33.622480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48469 (local address 127.0.98.62:38443)
0927 12:50:33.622605 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:33.622610 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:33.622896 (+ 286us) server_negotiation.cc:407] Connection header received
0927 12:50:33.623015 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:33.623017 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:33.623061 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:33.623127 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:33.623436 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:33.623963 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:33.625048 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:33.625301 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:33.625407 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:33.626310 (+ 903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:33.626313 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:33.626315 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:50:33.626327 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:33.626351 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:50:33.626360 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:33.626421 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:33.626558 (+ 137us) server_negotiation.cc:299] Negotiation successful
0927 12:50:33.626590 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 12:50:33.627259 22167 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a65c5b239e24b4492474bcae04bfe9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a65c5b239e24b4492474bcae04bfe9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38443 } }
I20260927 12:50:33.627641 22167 leader_election.cc:302] T 00000000000000000000000000000000 P 8a65c5b239e24b4492474bcae04bfe9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a65c5b239e24b4492474bcae04bfe9e; no voters:
I20260927 12:50:33.627962 22167 leader_election.cc:288] T 00000000000000000000000000000000 P 8a65c5b239e24b4492474bcae04bfe9e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:50:33.628062 22177 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a65c5b239e24b4492474bcae04bfe9e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:50:33.628268 22177 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a65c5b239e24b4492474bcae04bfe9e [term 1 LEADER]: Becoming Leader. State: Replica: 8a65c5b239e24b4492474bcae04bfe9e, State: Running, Role: LEADER
I20260927 12:50:33.628573 22177 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a65c5b239e24b4492474bcae04bfe9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a65c5b239e24b4492474bcae04bfe9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38443 } }
I20260927 12:50:33.629123 22167 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a65c5b239e24b4492474bcae04bfe9e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:50:33.630182 22178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a65c5b239e24b4492474bcae04bfe9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a65c5b239e24b4492474bcae04bfe9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a65c5b239e24b4492474bcae04bfe9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38443 } } }
I20260927 12:50:33.630313 22178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a65c5b239e24b4492474bcae04bfe9e [sys.catalog]: This master's current role is: LEADER
I20260927 12:50:33.630664 22179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a65c5b239e24b4492474bcae04bfe9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a65c5b239e24b4492474bcae04bfe9e. Latest consensus state: current_term: 1 leader_uuid: "8a65c5b239e24b4492474bcae04bfe9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a65c5b239e24b4492474bcae04bfe9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38443 } } }
I20260927 12:50:33.630816 22179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a65c5b239e24b4492474bcae04bfe9e [sys.catalog]: This master's current role is: LEADER
W20260927 12:50:33.634213 22192 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8a65c5b239e24b4492474bcae04bfe9e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 12:50:33.634279 22192 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 12:50:33.634341 22186 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:50:33.635031 22186 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:50:33.640396 22186 catalog_manager.cc:1347] Generated new cluster ID: 0644ee37e97240f4a2f7fbb94c67e3eb
I20260927 12:50:33.640524 22186 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:50:33.654384 22186 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:50:33.655443 22186 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:50:33.660776 22186 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a65c5b239e24b4492474bcae04bfe9e: Generated new TSK 0
I20260927 12:50:33.661532 22186 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:50:33.669616 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 22099
W20260927 12:50:33.674654 22084 connection.cc:535] client connection to 127.0.98.62:38443 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:38443): Connection reset by peer (error 104)
2026-09-27T12:50:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1326 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790513161160093-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:50:33 dist-test-slave-lkn4 krb5kdc[22198](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:50:33 dist-test-slave-lkn4 krb5kdc[22198](info): set up 2 sockets
Sep 27 12:50:33 dist-test-slave-lkn4 krb5kdc[22198](info): commencing operation
krb5kdc: starting...
W20260927 12:50:34.626089 392 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790513161160093-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790513161160093-392-0/krb5kdc/test-user.keytab.
Sep 27 12:50:34 dist-test-slave-lkn4 krb5kdc[22198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513434, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:50:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:50:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:50:34.745436 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790513161160093-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790513161160093-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790513161160093-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40015
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40015
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:50:34.855607 22211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:50:34.855908 22211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:50:34.855973 22211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:50:34.860049 22211 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:50:34.860121 22211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:50:34.860148 22211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:50:34.860170 22211 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:50:34.860198 22211 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:50:34.865767 22211 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790513161160093-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790513161160093-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40015
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790513161160093-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:50:34.866926 22211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:50:34.867964 22211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:50:34.876792 22217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:50:34.876822 22216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:50:34.876946 22211 server_base.cc:1034] running on GCE node
W20260927 12:50:34.876885 22219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:50:34.877662 22211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:50:34.878986 22211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:50:34.880177 22211 hybrid_clock.cc:648] HybridClock initialized: now 1790513434880152 us; error 36 us; skew 500 ppm
Sep 27 12:50:34 dist-test-slave-lkn4 krb5kdc[22198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513434, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:50:34.883085 22211 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:50:34.884008 22211 webserver.cc:460] Webserver started at http://127.0.98.62:41337/ using document root <none> and password file <none>
I20260927 12:50:34.884524 22211 fs_manager.cc:359] Metadata directory not provided
I20260927 12:50:34.884581 22211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:50:34.884754 22211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:50:34.886902 22211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790513161160093-392-0/minicluster-data/master-0/data/instance:
uuid: "97cce9ad619242208043bd66160cf268"
format_stamp: "Formatted at 2026-09-27 12:50:34 on dist-test-slave-lkn4"
server_key: "c45e286c3a0de59dd9e06bbf8e257e95"
server_key_iv: "6fc5c8793dd078bfc02f9e52ada9bc2f"
server_key_version: "encryptionkey@0"
I20260927 12:50:34.887418 22211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790513161160093-392-0/minicluster-data/master-0/wal/instance:
uuid: "97cce9ad619242208043bd66160cf268"
format_stamp: "Formatted at 2026-09-27 12:50:34 on dist-test-slave-lkn4"
server_key: "c45e286c3a0de59dd9e06bbf8e257e95"
server_key_iv: "6fc5c8793dd078bfc02f9e52ada9bc2f"
server_key_version: "encryptionkey@0"
I20260927 12:50:34.891208 22211 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 12:50:34.893668 22226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:34.894991 22211 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:50:34.895097 22211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790513161160093-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790513161160093-392-0/minicluster-data/master-0/wal
uuid: "97cce9ad619242208043bd66160cf268"
format_stamp: "Formatted at 2026-09-27 12:50:34 on dist-test-slave-lkn4"
server_key: "c45e286c3a0de59dd9e06bbf8e257e95"
server_key_iv: "6fc5c8793dd078bfc02f9e52ada9bc2f"
server_key_version: "encryptionkey@0"
I20260927 12:50:34.895213 22211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790513161160093-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790513161160093-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790513161160093-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:50:34.899937 22211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:50:34.902984 22211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:50:34.903131 22211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:50:34.909739 22211 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40015
I20260927 12:50:34.909749 22278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40015 every 8 connection(s)
I20260927 12:50:34.910689 22211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:50:34.911171 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 22211
I20260927 12:50:34.911331 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790513161160093-392-0/minicluster-data/master-0/wal/instance
I20260927 12:50:34.911587 392 external_mini_cluster.cc:1428] Setting key ee7402461027cfb7f3ca4195a40f54bf
I20260927 12:50:34.913714 22279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:50:34.916966 22282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:34.912923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34743 (local address 127.0.98.62:40015)
0927 12:50:34.913381 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:34.913397 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:34.913437 (+ 40us) server_negotiation.cc:407] Connection header received
0927 12:50:34.914194 (+ 757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:34.914214 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:34.914520 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:34.914881 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:34.915772 (+ 891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:34.915777 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:34.915790 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:50:34.915801 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:34.915859 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:50:34.915889 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:34.916028 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:34.916114 (+ 86us) server_negotiation.cc:299] Negotiation successful
0927 12:50:34.916268 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":139,"threads_started":1}
I20260927 12:50:34.920001 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:50:34.920020 22279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97cce9ad619242208043bd66160cf268: Bootstrap starting.
I20260927 12:50:34.922350 22279 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 97cce9ad619242208043bd66160cf268: Neither blocks nor log segments found. Creating new log.
I20260927 12:50:34.923225 22279 log.cc:824] T 00000000000000000000000000000000 P 97cce9ad619242208043bd66160cf268: Log is configured to *not* fsync() on all Append() calls
I20260927 12:50:34.923702 22282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:34.921945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56461 (local address 127.0.98.62:40015)
0927 12:50:34.922089 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:34.922094 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:34.922371 (+ 277us) server_negotiation.cc:407] Connection header received
0927 12:50:34.922489 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:34.922492 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:34.922531 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:34.922598 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:34.923332 (+ 734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:34.923336 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:34.923338 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:50:34.923349 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:34.923378 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:50:34.923389 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:34.923456 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:34.923543 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 12:50:34.923574 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 12:50:34.925760 22279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97cce9ad619242208043bd66160cf268: No bootstrap required, opened a new log
I20260927 12:50:34.929322 22279 raft_consensus.cc:348] T 00000000000000000000000000000000 P 97cce9ad619242208043bd66160cf268 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97cce9ad619242208043bd66160cf268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40015 } }
I20260927 12:50:34.929555 22279 raft_consensus.cc:374] T 00000000000000000000000000000000 P 97cce9ad619242208043bd66160cf268 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:50:34.929591 22279 raft_consensus.cc:729] T 00000000000000000000000000000000 P 97cce9ad619242208043bd66160cf268 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97cce9ad619242208043bd66160cf268, State: Initialized, Role: FOLLOWER
I20260927 12:50:34.930007 22279 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97cce9ad619242208043bd66160cf268 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97cce9ad619242208043bd66160cf268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40015 } }
I20260927 12:50:34.930173 22279 raft_consensus.cc:388] T 00000000000000000000000000000000 P 97cce9ad619242208043bd66160cf268 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:50:34.930238 22279 raft_consensus.cc:482] T 00000000000000000000000000000000 P 97cce9ad619242208043bd66160cf268 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:50:34.930333 22279 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 97cce9ad619242208043bd66160cf268 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:50:34.931243 22279 raft_consensus.cc:504] T 00000000000000000000000000000000 P 97cce9ad619242208043bd66160cf268 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97cce9ad619242208043bd66160cf268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40015 } }
I20260927 12:50:34.931560 22279 leader_election.cc:302] T 00000000000000000000000000000000 P 97cce9ad619242208043bd66160cf268 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 97cce9ad619242208043bd66160cf268; no voters:
I20260927 12:50:34.931861 22279 leader_election.cc:288] T 00000000000000000000000000000000 P 97cce9ad619242208043bd66160cf268 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:50:34.931965 22289 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 97cce9ad619242208043bd66160cf268 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:50:34.932179 22289 raft_consensus.cc:686] T 00000000000000000000000000000000 P 97cce9ad619242208043bd66160cf268 [term 1 LEADER]: Becoming Leader. State: Replica: 97cce9ad619242208043bd66160cf268, State: Running, Role: LEADER
I20260927 12:50:34.932487 22289 consensus_queue.cc:237] T 00000000000000000000000000000000 P 97cce9ad619242208043bd66160cf268 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97cce9ad619242208043bd66160cf268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40015 } }
I20260927 12:50:34.933065 22279 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97cce9ad619242208043bd66160cf268 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:50:34.934333 22291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97cce9ad619242208043bd66160cf268 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97cce9ad619242208043bd66160cf268. Latest consensus state: current_term: 1 leader_uuid: "97cce9ad619242208043bd66160cf268" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97cce9ad619242208043bd66160cf268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40015 } } }
I20260927 12:50:34.934363 22290 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97cce9ad619242208043bd66160cf268 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "97cce9ad619242208043bd66160cf268" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97cce9ad619242208043bd66160cf268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40015 } } }
I20260927 12:50:34.934471 22290 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97cce9ad619242208043bd66160cf268 [sys.catalog]: This master's current role is: LEADER
I20260927 12:50:34.934470 22291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97cce9ad619242208043bd66160cf268 [sys.catalog]: This master's current role is: LEADER
I20260927 12:50:34.934892 22298 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:50:34.937700 22298 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:50:34.942987 22298 catalog_manager.cc:1347] Generated new cluster ID: 1c8a0eff801f4465be569553a28e87ee
I20260927 12:50:34.943065 22298 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:50:34.954506 22298 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:50:34.954952 22298 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:50:34.962779 22298 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 97cce9ad619242208043bd66160cf268: Generated new TSK 0
I20260927 12:50:34.963743 22298 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:50:34.966852 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 22211
2026-09-27T12:50:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1297 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790513161160093-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:50:35 dist-test-slave-lkn4 krb5kdc[22310](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:50:35 dist-test-slave-lkn4 krb5kdc[22310](info): set up 2 sockets
Sep 27 12:50:35 dist-test-slave-lkn4 krb5kdc[22310](info): commencing operation
krb5kdc: starting...
W20260927 12:50:35.922226 392 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790513161160093-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790513161160093-392-0/krb5kdc/test-user.keytab.
Sep 27 12:50:36 dist-test-slave-lkn4 krb5kdc[22310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:50:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:50:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:50:36.042279 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790513161160093-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790513161160093-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790513161160093-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33071
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33071
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:50:36.151623 22323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:50:36.151923 22323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:50:36.151968 22323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:50:36.156038 22323 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:50:36.156111 22323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:50:36.156129 22323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:50:36.156145 22323 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:50:36.156159 22323 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:50:36.162024 22323 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790513161160093-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790513161160093-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33071
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790513161160093-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:50:36.163113 22323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:50:36.164122 22323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:50:36.172521 22328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:50:36.172549 22323 server_base.cc:1034] running on GCE node
W20260927 12:50:36.172498 22329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:50:36.172503 22331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:50:36.173301 22323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:50:36.174393 22323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:50:36.175545 22323 hybrid_clock.cc:648] HybridClock initialized: now 1790513436175512 us; error 44 us; skew 500 ppm
Sep 27 12:50:36 dist-test-slave-lkn4 krb5kdc[22310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513436, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:50:36.184396 22323 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:50:36.185489 22323 webserver.cc:460] Webserver started at http://127.0.98.62:41345/ using document root <none> and password file <none>
I20260927 12:50:36.186098 22323 fs_manager.cc:359] Metadata directory not provided
I20260927 12:50:36.186205 22323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:50:36.186450 22323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:50:36.188611 22323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790513161160093-392-0/minicluster-data/master-0/data/instance:
uuid: "e7ceef0b00c746b1af4cead0e25b8b62"
format_stamp: "Formatted at 2026-09-27 12:50:36 on dist-test-slave-lkn4"
server_key: "a7bf6bf87cc3fef78d93cfa7bc267156"
server_key_iv: "75fe957c901c941adb992feeaf0c1df3"
server_key_version: "encryptionkey@0"
I20260927 12:50:36.189172 22323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790513161160093-392-0/minicluster-data/master-0/wal/instance:
uuid: "e7ceef0b00c746b1af4cead0e25b8b62"
format_stamp: "Formatted at 2026-09-27 12:50:36 on dist-test-slave-lkn4"
server_key: "a7bf6bf87cc3fef78d93cfa7bc267156"
server_key_iv: "75fe957c901c941adb992feeaf0c1df3"
server_key_version: "encryptionkey@0"
I20260927 12:50:36.193069 22323 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 12:50:36.195704 22338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:36.196969 22323 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 12:50:36.197098 22323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790513161160093-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790513161160093-392-0/minicluster-data/master-0/wal
uuid: "e7ceef0b00c746b1af4cead0e25b8b62"
format_stamp: "Formatted at 2026-09-27 12:50:36 on dist-test-slave-lkn4"
server_key: "a7bf6bf87cc3fef78d93cfa7bc267156"
server_key_iv: "75fe957c901c941adb992feeaf0c1df3"
server_key_version: "encryptionkey@0"
I20260927 12:50:36.197216 22323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790513161160093-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790513161160093-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790513161160093-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:50:36.210572 22323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:50:36.213814 22323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:50:36.213995 22323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:50:36.220813 22323 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33071
I20260927 12:50:36.220839 22390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33071 every 8 connection(s)
I20260927 12:50:36.221822 22323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:50:36.224552 22391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:50:36.228794 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 22323
I20260927 12:50:36.228928 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790513161160093-392-0/minicluster-data/master-0/wal/instance
I20260927 12:50:36.229207 392 external_mini_cluster.cc:1428] Setting key 8d9541d256e9d4dda7b9e58d960c5b7c
I20260927 12:50:36.230500 22391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7ceef0b00c746b1af4cead0e25b8b62: Bootstrap starting.
I20260927 12:50:36.232755 22391 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7ceef0b00c746b1af4cead0e25b8b62: Neither blocks nor log segments found. Creating new log.
I20260927 12:50:36.233451 22391 log.cc:824] T 00000000000000000000000000000000 P e7ceef0b00c746b1af4cead0e25b8b62: Log is configured to *not* fsync() on all Append() calls
I20260927 12:50:36.235209 22391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7ceef0b00c746b1af4cead0e25b8b62: No bootstrap required, opened a new log
Sep 27 12:50:36 dist-test-slave-lkn4 krb5kdc[22310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:50:36.238880 22391 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7ceef0b00c746b1af4cead0e25b8b62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7ceef0b00c746b1af4cead0e25b8b62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33071 } }
I20260927 12:50:36.239171 22391 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7ceef0b00c746b1af4cead0e25b8b62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:50:36.239217 22391 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7ceef0b00c746b1af4cead0e25b8b62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7ceef0b00c746b1af4cead0e25b8b62, State: Initialized, Role: FOLLOWER
I20260927 12:50:36.239707 22391 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7ceef0b00c746b1af4cead0e25b8b62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7ceef0b00c746b1af4cead0e25b8b62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33071 } }
I20260927 12:50:36.239838 22391 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7ceef0b00c746b1af4cead0e25b8b62 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:50:36.239902 22391 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7ceef0b00c746b1af4cead0e25b8b62 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:50:36.239991 22391 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7ceef0b00c746b1af4cead0e25b8b62 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:50:36.240890 22391 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7ceef0b00c746b1af4cead0e25b8b62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7ceef0b00c746b1af4cead0e25b8b62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33071 } }
I20260927 12:50:36.241205 22391 leader_election.cc:302] T 00000000000000000000000000000000 P e7ceef0b00c746b1af4cead0e25b8b62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7ceef0b00c746b1af4cead0e25b8b62; no voters:
I20260927 12:50:36.241498 22391 leader_election.cc:288] T 00000000000000000000000000000000 P e7ceef0b00c746b1af4cead0e25b8b62 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:50:36.241602 22396 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7ceef0b00c746b1af4cead0e25b8b62 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:50:36.241809 22396 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7ceef0b00c746b1af4cead0e25b8b62 [term 1 LEADER]: Becoming Leader. State: Replica: e7ceef0b00c746b1af4cead0e25b8b62, State: Running, Role: LEADER
I20260927 12:50:36.242236 22396 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7ceef0b00c746b1af4cead0e25b8b62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7ceef0b00c746b1af4cead0e25b8b62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33071 } }
I20260927 12:50:36.242762 22391 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7ceef0b00c746b1af4cead0e25b8b62 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:50:36.243503 22396 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7ceef0b00c746b1af4cead0e25b8b62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7ceef0b00c746b1af4cead0e25b8b62. Latest consensus state: current_term: 1 leader_uuid: "e7ceef0b00c746b1af4cead0e25b8b62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7ceef0b00c746b1af4cead0e25b8b62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33071 } } }
I20260927 12:50:36.243676 22396 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7ceef0b00c746b1af4cead0e25b8b62 [sys.catalog]: This master's current role is: LEADER
I20260927 12:50:36.243907 22397 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7ceef0b00c746b1af4cead0e25b8b62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7ceef0b00c746b1af4cead0e25b8b62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7ceef0b00c746b1af4cead0e25b8b62" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33071 } } }
I20260927 12:50:36.244001 22397 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7ceef0b00c746b1af4cead0e25b8b62 [sys.catalog]: This master's current role is: LEADER
I20260927 12:50:36.244058 22405 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:50:36.244202 22394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:36.230626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54769 (local address 127.0.98.62:33071)
0927 12:50:36.231043 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:36.231060 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:36.231095 (+ 35us) server_negotiation.cc:407] Connection header received
0927 12:50:36.231753 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:36.231770 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:36.232116 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:36.232447 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:36.233212 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:36.234258 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:36.235094 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:36.235580 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:36.235721 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:36.238388 (+ 2667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:36.238393 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:36.238408 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:50:36.238440 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:36.240482 (+ 2042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:36.240922 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:36.240929 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:36.240935 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:36.241081 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:36.241343 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:36.241346 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:36.241348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:36.241913 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:36.242721 (+ 808us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:36.242869 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 12:50:36.243179 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":139,"threads_started":1}
I20260927 12:50:36.247242 22405 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:50:36.247326 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:50:36.252993 22405 catalog_manager.cc:1347] Generated new cluster ID: 2b06883535a64d29ad8798030bba34e3
I20260927 12:50:36.253057 22405 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:50:36.261183 22394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:36.250467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43221 (local address 127.0.98.62:33071)
0927 12:50:36.250586 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:36.250590 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:36.253089 (+ 2499us) server_negotiation.cc:407] Connection header received
0927 12:50:36.253271 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:36.253274 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:36.253321 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:36.253405 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:36.254252 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:36.255028 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:36.256124 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:36.256473 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:36.256550 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:36.258097 (+ 1547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:36.258101 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:36.258104 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:50:36.258133 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:36.259647 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:36.260078 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:36.260081 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:36.260083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:36.260158 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:36.260401 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:36.260403 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:36.260405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:36.260603 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:36.260709 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:36.260920 (+ 211us) server_negotiation.cc:299] Negotiation successful
0927 12:50:36.261023 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 12:50:36.269788 22405 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:50:36.270816 22405 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:50:36.287607 22405 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7ceef0b00c746b1af4cead0e25b8b62: Generated new TSK 0
I20260927 12:50:36.288371 22405 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:50:36.293859 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 22323
2026-09-27T12:50:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1326 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790513161160093-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:50:36 dist-test-slave-lkn4 krb5kdc[22422](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:50:36 dist-test-slave-lkn4 krb5kdc[22422](info): set up 2 sockets
Sep 27 12:50:36 dist-test-slave-lkn4 krb5kdc[22422](info): commencing operation
krb5kdc: starting...
W20260927 12:50:37.246075 392 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790513161160093-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790513161160093-392-0/krb5kdc/test-user.keytab.
Sep 27 12:50:37 dist-test-slave-lkn4 krb5kdc[22422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:50:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:50:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:50:37.365281 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790513161160093-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790513161160093-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790513161160093-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33251
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33251
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:50:37.475695 22435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:50:37.476013 22435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:50:37.476073 22435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:50:37.480253 22435 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:50:37.480340 22435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:50:37.480367 22435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:50:37.480389 22435 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:50:37.480409 22435 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:50:37.486104 22435 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790513161160093-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790513161160093-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33251
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790513161160093-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:50:37.487257 22435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:50:37.488309 22435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:50:37.497129 22443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:50:37.497191 22441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:50:37.497121 22440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:50:37.497411 22435 server_base.cc:1034] running on GCE node
I20260927 12:50:37.498202 22435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:50:37.499351 22435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:50:37.500515 22435 hybrid_clock.cc:648] HybridClock initialized: now 1790513437500490 us; error 36 us; skew 500 ppm
Sep 27 12:50:37 dist-test-slave-lkn4 krb5kdc[22422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513437, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:50:37.503207 22435 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:50:37.504097 22435 webserver.cc:460] Webserver started at http://127.0.98.62:37959/ using document root <none> and password file <none>
I20260927 12:50:37.504655 22435 fs_manager.cc:359] Metadata directory not provided
I20260927 12:50:37.504724 22435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:50:37.504928 22435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:50:37.507045 22435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790513161160093-392-0/minicluster-data/master-0/data/instance:
uuid: "63fd5d11c13d40479b2253c34b61fe0b"
format_stamp: "Formatted at 2026-09-27 12:50:37 on dist-test-slave-lkn4"
server_key: "4ee7f4d3067942117dc3f57e0c59e6b1"
server_key_iv: "01e9c0f06b743b89a80ca65bafd93eb4"
server_key_version: "encryptionkey@0"
I20260927 12:50:37.507587 22435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790513161160093-392-0/minicluster-data/master-0/wal/instance:
uuid: "63fd5d11c13d40479b2253c34b61fe0b"
format_stamp: "Formatted at 2026-09-27 12:50:37 on dist-test-slave-lkn4"
server_key: "4ee7f4d3067942117dc3f57e0c59e6b1"
server_key_iv: "01e9c0f06b743b89a80ca65bafd93eb4"
server_key_version: "encryptionkey@0"
I20260927 12:50:37.511406 22435 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 12:50:37.514017 22450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:37.515317 22435 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:50:37.515436 22435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790513161160093-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790513161160093-392-0/minicluster-data/master-0/wal
uuid: "63fd5d11c13d40479b2253c34b61fe0b"
format_stamp: "Formatted at 2026-09-27 12:50:37 on dist-test-slave-lkn4"
server_key: "4ee7f4d3067942117dc3f57e0c59e6b1"
server_key_iv: "01e9c0f06b743b89a80ca65bafd93eb4"
server_key_version: "encryptionkey@0"
I20260927 12:50:37.515542 22435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790513161160093-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790513161160093-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790513161160093-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:50:37.531010 22435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:50:37.534444 22435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:50:37.534631 22435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:50:37.541595 22435 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33251
I20260927 12:50:37.541622 22502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33251 every 8 connection(s)
I20260927 12:50:37.542596 22435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:50:37.545574 22503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:50:37.551294 22503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63fd5d11c13d40479b2253c34b61fe0b: Bootstrap starting.
I20260927 12:50:37.551553 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 22435
I20260927 12:50:37.551657 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790513161160093-392-0/minicluster-data/master-0/wal/instance
I20260927 12:50:37.551918 392 external_mini_cluster.cc:1428] Setting key 64cddef92c53683b57e9df542673cc9b
I20260927 12:50:37.554026 22503 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63fd5d11c13d40479b2253c34b61fe0b: Neither blocks nor log segments found. Creating new log.
I20260927 12:50:37.554780 22503 log.cc:824] T 00000000000000000000000000000000 P 63fd5d11c13d40479b2253c34b61fe0b: Log is configured to *not* fsync() on all Append() calls
I20260927 12:50:37.556656 22503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63fd5d11c13d40479b2253c34b61fe0b: No bootstrap required, opened a new log
I20260927 12:50:37.560214 22503 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63fd5d11c13d40479b2253c34b61fe0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63fd5d11c13d40479b2253c34b61fe0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33251 } }
I20260927 12:50:37.560442 22503 raft_consensus.cc:374] T 00000000000000000000000000000000 P 63fd5d11c13d40479b2253c34b61fe0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:50:37.560498 22503 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63fd5d11c13d40479b2253c34b61fe0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63fd5d11c13d40479b2253c34b61fe0b, State: Initialized, Role: FOLLOWER
I20260927 12:50:37.560850 22506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:37.553243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46045 (local address 127.0.98.62:33251)
0927 12:50:37.553797 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:37.553815 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:37.553854 (+ 39us) server_negotiation.cc:407] Connection header received
0927 12:50:37.554594 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:37.554616 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:37.555010 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:37.555336 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:37.555757 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:37.556759 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:37.557650 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:37.558174 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:37.558320 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:37.559117 (+ 797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:37.559129 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:37.559144 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:50:37.559538 (+ 394us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:37.559591 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:50:37.559614 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:37.559699 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:37.559841 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 12:50:37.560057 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":151,"threads_started":1}
I20260927 12:50:37.560945 22503 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63fd5d11c13d40479b2253c34b61fe0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63fd5d11c13d40479b2253c34b61fe0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33251 } }
I20260927 12:50:37.561102 22503 raft_consensus.cc:388] T 00000000000000000000000000000000 P 63fd5d11c13d40479b2253c34b61fe0b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:50:37.561167 22503 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63fd5d11c13d40479b2253c34b61fe0b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:50:37.561268 22503 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63fd5d11c13d40479b2253c34b61fe0b [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:50:37.562326 22503 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63fd5d11c13d40479b2253c34b61fe0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63fd5d11c13d40479b2253c34b61fe0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33251 } }
I20260927 12:50:37.562664 22503 leader_election.cc:302] T 00000000000000000000000000000000 P 63fd5d11c13d40479b2253c34b61fe0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63fd5d11c13d40479b2253c34b61fe0b; no voters:
I20260927 12:50:37.563030 22503 leader_election.cc:288] T 00000000000000000000000000000000 P 63fd5d11c13d40479b2253c34b61fe0b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:50:37.563100 22508 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63fd5d11c13d40479b2253c34b61fe0b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:50:37.563314 22508 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63fd5d11c13d40479b2253c34b61fe0b [term 1 LEADER]: Becoming Leader. State: Replica: 63fd5d11c13d40479b2253c34b61fe0b, State: Running, Role: LEADER
I20260927 12:50:37.563702 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:50:37.563691 22508 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63fd5d11c13d40479b2253c34b61fe0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63fd5d11c13d40479b2253c34b61fe0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33251 } }
I20260927 12:50:37.564299 22503 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63fd5d11c13d40479b2253c34b61fe0b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:50:37.565172 22509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63fd5d11c13d40479b2253c34b61fe0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63fd5d11c13d40479b2253c34b61fe0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63fd5d11c13d40479b2253c34b61fe0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33251 } } }
I20260927 12:50:37.565308 22509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63fd5d11c13d40479b2253c34b61fe0b [sys.catalog]: This master's current role is: LEADER
I20260927 12:50:37.565596 22510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63fd5d11c13d40479b2253c34b61fe0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63fd5d11c13d40479b2253c34b61fe0b. Latest consensus state: current_term: 1 leader_uuid: "63fd5d11c13d40479b2253c34b61fe0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63fd5d11c13d40479b2253c34b61fe0b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33251 } } }
I20260927 12:50:37.565670 22510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63fd5d11c13d40479b2253c34b61fe0b [sys.catalog]: This master's current role is: LEADER
I20260927 12:50:37.566015 22521 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:50:37.569401 22521 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:50:37.570705 22506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:37.567051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47861 (local address 127.0.98.62:33251)
0927 12:50:37.567161 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:37.567166 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:37.567424 (+ 258us) server_negotiation.cc:407] Connection header received
0927 12:50:37.567537 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:37.567540 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:37.567583 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:37.567643 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:37.567895 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:37.568397 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:37.569207 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:37.569426 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:37.569510 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:37.570323 (+ 813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:37.570326 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:37.570329 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:50:37.570341 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:37.570365 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:50:37.570375 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:37.570432 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:37.570524 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 12:50:37.570572 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260927 12:50:37.575183 22521 catalog_manager.cc:1347] Generated new cluster ID: 4f53cb9b72424573a71c2e4f65f2a837
I20260927 12:50:37.575259 22521 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:50:37.598215 22521 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:50:37.599268 22521 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:50:37.610592 22521 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 63fd5d11c13d40479b2253c34b61fe0b: Generated new TSK 0
I20260927 12:50:37.611497 22521 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:50:37.623171 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 22435
2026-09-27T12:50:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1329 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790513161160093-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:50:37 dist-test-slave-lkn4 krb5kdc[22534](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:50:37 dist-test-slave-lkn4 krb5kdc[22534](info): set up 2 sockets
Sep 27 12:50:37 dist-test-slave-lkn4 krb5kdc[22534](info): commencing operation
krb5kdc: starting...
W20260927 12:50:38.582154 392 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790513161160093-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790513161160093-392-0/krb5kdc/test-user.keytab.
Sep 27 12:50:38 dist-test-slave-lkn4 krb5kdc[22534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:50:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:50:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:50:38.705344 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790513161160093-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790513161160093-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790513161160093-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42097
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42097
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:50:38.815049 22547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:50:38.815346 22547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:50:38.815389 22547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:50:38.819720 22547 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:50:38.819801 22547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:50:38.819821 22547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:50:38.819836 22547 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:50:38.819850 22547 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:50:38.825332 22547 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790513161160093-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790513161160093-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42097
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790513161160093-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:50:38.826514 22547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:50:38.827545 22547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:50:38.836251 22552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:50:38.836306 22547 server_base.cc:1034] running on GCE node
W20260927 12:50:38.836251 22553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:50:38.836265 22555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:50:38.837116 22547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:50:38.838351 22547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:50:38.839524 22547 hybrid_clock.cc:648] HybridClock initialized: now 1790513438839490 us; error 45 us; skew 500 ppm
Sep 27 12:50:38 dist-test-slave-lkn4 krb5kdc[22534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513438, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:50:38.842475 22547 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:50:38.843431 22547 webserver.cc:460] Webserver started at http://127.0.98.62:38875/ using document root <none> and password file <none>
I20260927 12:50:38.843972 22547 fs_manager.cc:359] Metadata directory not provided
I20260927 12:50:38.844038 22547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:50:38.844239 22547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:50:38.846418 22547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790513161160093-392-0/minicluster-data/master-0/data/instance:
uuid: "61a88da71070461eac89397c8cd208b8"
format_stamp: "Formatted at 2026-09-27 12:50:38 on dist-test-slave-lkn4"
server_key: "b5037085b9b124a514025ac4cce04f15"
server_key_iv: "b8129c7a064849747249ddc10898c081"
server_key_version: "encryptionkey@0"
I20260927 12:50:38.847100 22547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790513161160093-392-0/minicluster-data/master-0/wal/instance:
uuid: "61a88da71070461eac89397c8cd208b8"
format_stamp: "Formatted at 2026-09-27 12:50:38 on dist-test-slave-lkn4"
server_key: "b5037085b9b124a514025ac4cce04f15"
server_key_iv: "b8129c7a064849747249ddc10898c081"
server_key_version: "encryptionkey@0"
I20260927 12:50:38.851516 22547 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260927 12:50:38.853957 22562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:38.855358 22547 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:50:38.855485 22547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790513161160093-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790513161160093-392-0/minicluster-data/master-0/wal
uuid: "61a88da71070461eac89397c8cd208b8"
format_stamp: "Formatted at 2026-09-27 12:50:38 on dist-test-slave-lkn4"
server_key: "b5037085b9b124a514025ac4cce04f15"
server_key_iv: "b8129c7a064849747249ddc10898c081"
server_key_version: "encryptionkey@0"
I20260927 12:50:38.855599 22547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790513161160093-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790513161160093-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790513161160093-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:50:38.860162 22547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:50:38.863252 22547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:50:38.863440 22547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:50:38.870375 22547 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42097
I20260927 12:50:38.870424 22614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42097 every 8 connection(s)
I20260927 12:50:38.871335 22547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:50:38.871521 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 22547
I20260927 12:50:38.871656 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790513161160093-392-0/minicluster-data/master-0/wal/instance
I20260927 12:50:38.872016 392 external_mini_cluster.cc:1428] Setting key 9f295aaf939b0e8f3e2870eee6ca653f
I20260927 12:50:38.874552 22615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:50:38.877252 22618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:38.873371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35961 (local address 127.0.98.62:42097)
0927 12:50:38.873760 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:38.873776 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:38.873812 (+ 36us) server_negotiation.cc:407] Connection header received
0927 12:50:38.874482 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:38.874500 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:38.874829 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:38.875129 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:38.875966 (+ 837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:38.875972 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:38.875984 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:50:38.875995 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:38.876048 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:50:38.876068 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:38.876183 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:38.876280 (+ 97us) server_negotiation.cc:299] Negotiation successful
0927 12:50:38.876427 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":102,"threads_started":1}
I20260927 12:50:38.880362 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:50:38.880841 22615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61a88da71070461eac89397c8cd208b8: Bootstrap starting.
I20260927 12:50:38.883147 22615 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61a88da71070461eac89397c8cd208b8: Neither blocks nor log segments found. Creating new log.
I20260927 12:50:38.883893 22618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:38.882346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46083 (local address 127.0.98.62:42097)
0927 12:50:38.882468 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:38.882473 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:38.882645 (+ 172us) server_negotiation.cc:407] Connection header received
0927 12:50:38.882768 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:38.882772 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:38.882804 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:38.882865 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:38.883537 (+ 672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:38.883540 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:38.883543 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:50:38.883554 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:38.883581 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:50:38.883590 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:38.883658 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:38.883740 (+ 82us) server_negotiation.cc:299] Negotiation successful
0927 12:50:38.883777 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 12:50:38.883885 22615 log.cc:824] T 00000000000000000000000000000000 P 61a88da71070461eac89397c8cd208b8: Log is configured to *not* fsync() on all Append() calls
I20260927 12:50:38.886257 22615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61a88da71070461eac89397c8cd208b8: No bootstrap required, opened a new log
I20260927 12:50:38.889887 22615 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61a88da71070461eac89397c8cd208b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61a88da71070461eac89397c8cd208b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42097 } }
I20260927 12:50:38.890190 22615 raft_consensus.cc:374] T 00000000000000000000000000000000 P 61a88da71070461eac89397c8cd208b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:50:38.890254 22615 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61a88da71070461eac89397c8cd208b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61a88da71070461eac89397c8cd208b8, State: Initialized, Role: FOLLOWER
I20260927 12:50:38.890636 22615 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61a88da71070461eac89397c8cd208b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61a88da71070461eac89397c8cd208b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42097 } }
I20260927 12:50:38.890745 22615 raft_consensus.cc:388] T 00000000000000000000000000000000 P 61a88da71070461eac89397c8cd208b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:50:38.890790 22615 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61a88da71070461eac89397c8cd208b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:50:38.890858 22615 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61a88da71070461eac89397c8cd208b8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:50:38.891712 22615 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61a88da71070461eac89397c8cd208b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61a88da71070461eac89397c8cd208b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42097 } }
I20260927 12:50:38.892030 22615 leader_election.cc:302] T 00000000000000000000000000000000 P 61a88da71070461eac89397c8cd208b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61a88da71070461eac89397c8cd208b8; no voters:
I20260927 12:50:38.892311 22615 leader_election.cc:288] T 00000000000000000000000000000000 P 61a88da71070461eac89397c8cd208b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:50:38.892423 22625 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61a88da71070461eac89397c8cd208b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:50:38.892668 22625 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61a88da71070461eac89397c8cd208b8 [term 1 LEADER]: Becoming Leader. State: Replica: 61a88da71070461eac89397c8cd208b8, State: Running, Role: LEADER
I20260927 12:50:38.892961 22625 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61a88da71070461eac89397c8cd208b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61a88da71070461eac89397c8cd208b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42097 } }
I20260927 12:50:38.893476 22615 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61a88da71070461eac89397c8cd208b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:50:38.894666 22626 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61a88da71070461eac89397c8cd208b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "61a88da71070461eac89397c8cd208b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61a88da71070461eac89397c8cd208b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42097 } } }
I20260927 12:50:38.894809 22626 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61a88da71070461eac89397c8cd208b8 [sys.catalog]: This master's current role is: LEADER
I20260927 12:50:38.894666 22627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61a88da71070461eac89397c8cd208b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61a88da71070461eac89397c8cd208b8. Latest consensus state: current_term: 1 leader_uuid: "61a88da71070461eac89397c8cd208b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61a88da71070461eac89397c8cd208b8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42097 } } }
I20260927 12:50:38.894912 22627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61a88da71070461eac89397c8cd208b8 [sys.catalog]: This master's current role is: LEADER
W20260927 12:50:38.898312 22640 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 61a88da71070461eac89397c8cd208b8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 12:50:38.898377 22640 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 12:50:38.898440 22635 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:50:38.899151 22635 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:50:38.904147 22635 catalog_manager.cc:1347] Generated new cluster ID: 207c8e218fda4c68b47484fc489c8f9a
I20260927 12:50:38.904213 22635 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:50:38.912235 22635 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:50:38.912582 22635 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:50:38.918737 22635 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 61a88da71070461eac89397c8cd208b8: Generated new TSK 0
I20260927 12:50:38.919562 22635 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:50:38.928377 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 22547
2026-09-27T12:50:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1304 ms)
[----------] 18 tests from AuthTokenIssuingTest (23934 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:50:38 dist-test-slave-lkn4 krb5kdc[22646](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:50:38 dist-test-slave-lkn4 krb5kdc[22646](info): set up 2 sockets
Sep 27 12:50:38 dist-test-slave-lkn4 krb5kdc[22646](info): commencing operation
krb5kdc: starting...
W20260927 12:50:39.890170 392 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/krb5kdc/test-user.keytab.
Sep 27 12:50:39 dist-test-slave-lkn4 krb5kdc[22646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:50:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:50:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:50:40.008929 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42185
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42185,127.0.98.61:38455,127.0.98.60:35981
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:50:40.119835 22659 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 12:50:40.120167 22659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:50:40.120204 22659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:50:40.120270 22659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:50:40.124358 22659 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:50:40.124434 22659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:50:40.124464 22659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:50:40.124483 22659 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:50:40.124504 22659 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:50:40.129985 22659 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42185,127.0.98.61:38455,127.0.98.60:35981
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22659
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:50:40.131178 22659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:50:40.132336 22659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:50:40.141096 22665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:50:40.141099 22667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:50:40.141135 22664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:50:40.141414 22659 server_base.cc:1034] running on GCE node
I20260927 12:50:40.142031 22659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:50:40.143249 22659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:50:40.144434 22659 hybrid_clock.cc:648] HybridClock initialized: now 1790513440144409 us; error 42 us; skew 500 ppm
Sep 27 12:50:40 dist-test-slave-lkn4 krb5kdc[22646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:50:40.147416 22659 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:50:40.148376 22659 webserver.cc:460] Webserver started at http://127.0.98.62:39315/ using document root <none> and password file <none>
I20260927 12:50:40.148926 22659 fs_manager.cc:359] Metadata directory not provided
I20260927 12:50:40.148981 22659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:50:40.149152 22659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:50:40.151232 22659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/minicluster-data/master-0/data/instance:
uuid: "976518232da04ad4ab0a261ca6ca25d1"
format_stamp: "Formatted at 2026-09-27 12:50:40 on dist-test-slave-lkn4"
server_key: "a0bf7cdc7578a29672694d0e24751303"
server_key_iv: "cef85e4c1a73414639e6fdb0420dba44"
server_key_version: "encryptionkey@0"
I20260927 12:50:40.151746 22659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/minicluster-data/master-0/wal/instance:
uuid: "976518232da04ad4ab0a261ca6ca25d1"
format_stamp: "Formatted at 2026-09-27 12:50:40 on dist-test-slave-lkn4"
server_key: "a0bf7cdc7578a29672694d0e24751303"
server_key_iv: "cef85e4c1a73414639e6fdb0420dba44"
server_key_version: "encryptionkey@0"
I20260927 12:50:40.155467 22659 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 12:50:40.157915 22674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:40.159186 22659 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:50:40.159289 22659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/minicluster-data/master-0/wal
uuid: "976518232da04ad4ab0a261ca6ca25d1"
format_stamp: "Formatted at 2026-09-27 12:50:40 on dist-test-slave-lkn4"
server_key: "a0bf7cdc7578a29672694d0e24751303"
server_key_iv: "cef85e4c1a73414639e6fdb0420dba44"
server_key_version: "encryptionkey@0"
I20260927 12:50:40.159396 22659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:50:40.173084 22659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:50:40.176213 22659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:50:40.176386 22659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:50:40.183228 22659 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42185
I20260927 12:50:40.183560 22726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42185 every 8 connection(s)
I20260927 12:50:40.185096 22659 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:50:40.188448 22727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:50:40.192592 22727 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42185 } has no permanent_uuid. Determining permanent_uuid...
I20260927 12:50:40.195399 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 22659
I20260927 12:50:40.195536 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/minicluster-data/master-0/wal/instance
I20260927 12:50:40.195852 392 external_mini_cluster.cc:1428] Setting key 8a9556f65f5288bc584367240e5f3929
Sep 27 12:50:40 dist-test-slave-lkn4 krb5kdc[22646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260927 12:50:40.207037 22729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:40.194884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41932 (local address 127.0.98.62:42185)
0927 12:50:40.195337 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:40.195354 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:40.195590 (+ 236us) server_negotiation.cc:407] Connection header received
0927 12:50:40.196685 (+ 1095us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:40.196689 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:40.196872 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:40.197032 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:40.197884 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:40.198773 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:40.199736 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:40.200102 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:40.200262 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:40.202450 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:40.202455 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:40.202466 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:50:40.202486 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:40.203963 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:40.204405 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:40.204410 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:40.204411 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:40.204511 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:40.204779 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:40.204782 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:40.204784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:40.205172 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:40.205298 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:40.205659 (+ 361us) server_negotiation.cc:299] Negotiation successful
0927 12:50:40.205926 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":305,"thread_start_us":137,"threads_started":1}
I20260927 12:50:40.207037 22730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:40.194884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42185 (local address 127.0.0.1:41932)
0927 12:50:40.195449 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:40.195495 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 12:50:40.196377 (+ 882us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:50:40.197042 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:50:40.197047 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:50:40.197486 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:50:40.197735 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:40.197757 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:40.198885 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:40.198887 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:40.199623 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:40.199632 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:40.200242 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:40.200246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:40.200350 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:40.200889 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:50:40.200905 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:50:40.202320 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:50:40.204065 (+ 1745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:50:40.204073 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:50:40.204084 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:50:40.204307 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:50:40.204590 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:50:40.204593 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:50:40.204594 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:50:40.204701 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:50:40.205274 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:50:40.205280 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:50:40.205414 (+ 134us) client_negotiation.cc:769] Sending connection context
0927 12:50:40.205562 (+ 148us) client_negotiation.cc:240] Negotiation successful
0927 12:50:40.205980 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"mutex_wait_us":111,"spinlock_wait_cycles":56064,"thread_start_us":175,"threads_started":1}
I20260927 12:50:40.209619 22727 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38455 } has no permanent_uuid. Determining permanent_uuid...
I20260927 12:50:40.210404 22730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:40.210152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38455 (local address 127.0.0.1:37170)
0927 12:50:40.210262 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:40.210339 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 12:50:40.210628 22676 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38455: connect: Connection refused (error 111)
I20260927 12:50:40.211493 22730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:40.211299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38455 (local address 127.0.0.1:37178)
0927 12:50:40.211397 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:40.211440 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 12:50:40.211624 22727 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:38455: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38455: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 12:50:40.220921 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:38455
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42185,127.0.98.61:38455,127.0.98.60:35981
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 12:50:40.260865 22727 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38455 } attempt: 1
I20260927 12:50:40.261646 22730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:40.261374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38455 (local address 127.0.0.1:37180)
0927 12:50:40.261508 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:40.261568 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 12:50:40.262456 22730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:40.262219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38455 (local address 127.0.0.1:37188)
0927 12:50:40.262336 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:40.262391 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 12:50:40.262598 22727 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:38455: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38455: connect: Connection refused (error 111)
I20260927 12:50:40.330746 22727 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38455 } attempt: 2
W20260927 12:50:40.330865 22734 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 12:50:40.331199 22734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:50:40.331241 22734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:50:40.331308 22734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 12:50:40.331609 22730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:40.331286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38455 (local address 127.0.0.1:37196)
0927 12:50:40.331451 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:40.331522 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 12:50:40.332309 22730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:40.332096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38455 (local address 127.0.0.1:37198)
0927 12:50:40.332197 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:40.332248 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 12:50:40.332458 22727 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:38455: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38455: connect: Connection refused (error 111)
W20260927 12:50:40.335651 22734 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:50:40.335703 22734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:50:40.335722 22734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:50:40.335740 22734 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:50:40.335768 22734 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:50:40.341077 22734 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42185,127.0.98.61:38455,127.0.98.60:35981
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:38455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:50:40.342167 22734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:50:40.343186 22734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:50:40.352133 22739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:50:40.352156 22742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:50:40.352146 22740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:50:40.352212 22734 server_base.cc:1034] running on GCE node
I20260927 12:50:40.353039 22734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:50:40.354249 22734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:50:40.355427 22734 hybrid_clock.cc:648] HybridClock initialized: now 1790513440355391 us; error 48 us; skew 500 ppm
Sep 27 12:50:40 dist-test-slave-lkn4 krb5kdc[22646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:50:40.358284 22734 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 12:50:40.359222 22734 webserver.cc:460] Webserver started at http://127.0.98.61:38651/ using document root <none> and password file <none>
I20260927 12:50:40.359760 22734 fs_manager.cc:359] Metadata directory not provided
I20260927 12:50:40.359825 22734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:50:40.360008 22734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:50:40.362097 22734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/minicluster-data/master-1/data/instance:
uuid: "5e8e89de53064882b7c16ebd1791897d"
format_stamp: "Formatted at 2026-09-27 12:50:40 on dist-test-slave-lkn4"
server_key: "aa1aa727638b99932ed96ce984ec8fb8"
server_key_iv: "3aa8fb402a6175906563390e9d69f0d1"
server_key_version: "encryptionkey@0"
I20260927 12:50:40.362634 22734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/minicluster-data/master-1/wal/instance:
uuid: "5e8e89de53064882b7c16ebd1791897d"
format_stamp: "Formatted at 2026-09-27 12:50:40 on dist-test-slave-lkn4"
server_key: "aa1aa727638b99932ed96ce984ec8fb8"
server_key_iv: "3aa8fb402a6175906563390e9d69f0d1"
server_key_version: "encryptionkey@0"
I20260927 12:50:40.366257 22734 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 12:50:40.368651 22749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:40.369877 22734 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:50:40.369999 22734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/minicluster-data/master-1/wal
uuid: "5e8e89de53064882b7c16ebd1791897d"
format_stamp: "Formatted at 2026-09-27 12:50:40 on dist-test-slave-lkn4"
server_key: "aa1aa727638b99932ed96ce984ec8fb8"
server_key_iv: "3aa8fb402a6175906563390e9d69f0d1"
server_key_version: "encryptionkey@0"
I20260927 12:50:40.370159 22734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:50:40.397143 22734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:50:40.400612 22734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:50:40.400820 22734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:50:40.408250 22734 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:38455
I20260927 12:50:40.408509 22801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:38455 every 8 connection(s)
I20260927 12:50:40.409296 22734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/minicluster-data/master-1/data/info.pb
I20260927 12:50:40.412081 22802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:50:40.415799 22802 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42185 } has no permanent_uuid. Determining permanent_uuid...
I20260927 12:50:40.417346 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 22734
I20260927 12:50:40.417438 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/minicluster-data/master-1/wal/instance
I20260927 12:50:40.417683 392 external_mini_cluster.cc:1428] Setting key 80308d0d49a1b3b904f346c3aec6a592
I20260927 12:50:40.423620 22727 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38455 } attempt: 3
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Sep 27 12:50:40 dist-test-slave-lkn4 krb5kdc[22646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 27 12:50:40 dist-test-slave-lkn4 krb5kdc[22646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 12:50:40.431449 22729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:40.417769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41948 (local address 127.0.98.62:42185)
0927 12:50:40.417925 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:40.417930 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:40.418832 (+ 902us) server_negotiation.cc:407] Connection header received
0927 12:50:40.419899 (+ 1067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:40.419903 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:40.419968 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:40.420076 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:40.421257 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:40.422103 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:40.423246 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:40.423657 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:40.423777 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:40.426644 (+ 2867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:40.426648 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:40.426651 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:50:40.426681 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:40.428317 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:40.428974 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:40.428977 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:40.428979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:40.429068 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:40.429454 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:40.429457 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:40.429458 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:40.429660 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:40.429745 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:40.431141 (+ 1396us) server_negotiation.cc:299] Negotiation successful
0927 12:50:40.431288 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 12:50:40.431667 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:40.418118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42185 (local address 127.0.0.1:41948)
0927 12:50:40.418627 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:40.418687 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 12:50:40.419665 (+ 978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:50:40.420210 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:50:40.420217 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:50:40.420605 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:50:40.421090 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:40.421104 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:40.422241 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:40.422244 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:40.423120 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:40.423129 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:40.423859 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:40.423864 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:40.424046 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:40.424752 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:50:40.424773 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:50:40.426496 (+ 1723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:50:40.428427 (+ 1931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:50:40.428434 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:50:40.428450 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:50:40.428874 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:50:40.429163 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:50:40.429167 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:50:40.429169 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:50:40.429360 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:50:40.429818 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:50:40.429825 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:50:40.430219 (+ 394us) client_negotiation.cc:769] Sending connection context
0927 12:50:40.430479 (+ 260us) client_negotiation.cc:240] Negotiation successful
0927 12:50:40.430804 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"spinlock_wait_cycles":2816,"thread_start_us":131,"threads_started":1}
I20260927 12:50:40.433320 22802 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38455 } has no permanent_uuid. Determining permanent_uuid...
I20260927 12:50:40.434922 22730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:40.424189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38455 (local address 127.0.0.1:37208)
0927 12:50:40.424396 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:40.424414 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 12:50:40.424497 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:50:40.425214 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:50:40.425218 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:50:40.425403 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:50:40.425541 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:40.425549 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:40.426868 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:40.426872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:40.427447 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:40.427456 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:40.428146 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:40.428149 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:40.428243 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:40.428602 (+ 359us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:50:40.428615 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:50:40.429952 (+ 1337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:50:40.433028 (+ 3076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:50:40.433032 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:50:40.433034 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:50:40.433283 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:50:40.433571 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:50:40.433574 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:50:40.433576 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:50:40.433668 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:50:40.434402 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:50:40.434405 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:50:40.434506 (+ 101us) client_negotiation.cc:769] Sending connection context
0927 12:50:40.434618 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 12:50:40.434762 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128}
I20260927 12:50:40.435031 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:40.424248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37208 (local address 127.0.98.61:38455)
0927 12:50:40.424710 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:40.424722 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:40.424745 (+ 23us) server_negotiation.cc:407] Connection header received
0927 12:50:40.424807 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:40.424812 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:40.425029 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:40.425198 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:40.425685 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:40.426761 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:40.427549 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:40.428031 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:40.428145 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:40.430103 (+ 1958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:40.430114 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:40.430129 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:50:40.430155 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:40.432923 (+ 2768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:40.433394 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:40.433400 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:40.433401 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:40.433488 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:40.433759 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:40.433762 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:40.433764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:40.434282 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:40.434404 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:40.434703 (+ 299us) server_negotiation.cc:299] Negotiation successful
0927 12:50:40.434863 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":148,"threads_started":1}
I20260927 12:50:40.437009 22727 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35981 } has no permanent_uuid. Determining permanent_uuid...
I20260927 12:50:40.437685 22730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:40.437445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35981 (local address 127.0.0.1:56416)
0927 12:50:40.437553 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:40.437608 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260927 12:50:40.438409 22730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:40.438144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35981 (local address 127.0.0.1:56424)
0927 12:50:40.438310 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:40.438357 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260927 12:50:40.438589 22727 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35981: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35981: connect: Connection refused (error 111)
Sep 27 12:50:40 dist-test-slave-lkn4 krb5kdc[22646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 12:50:40.444787 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:40.433796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38455 (local address 127.0.0.1:37216)
0927 12:50:40.433935 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:40.433950 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:50:40.434026 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:50:40.434504 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:50:40.434507 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:50:40.434668 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:50:40.434789 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:40.434796 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:40.435736 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:40.435740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:40.436374 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:40.436383 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:40.436983 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:40.436986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:40.437082 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:40.437566 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:50:40.437579 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:50:40.441289 (+ 3710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:50:40.443440 (+ 2151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:50:40.443443 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:50:40.443446 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:50:40.443620 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:50:40.443863 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:50:40.443866 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:50:40.443868 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:50:40.443940 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:50:40.444275 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:50:40.444279 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:50:40.444362 (+ 83us) client_negotiation.cc:769] Sending connection context
0927 12:50:40.444443 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 12:50:40.444605 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 12:50:40.444816 22808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:40.434002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37216 (local address 127.0.98.61:38455)
0927 12:50:40.434311 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:40.434320 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:40.434332 (+ 12us) server_negotiation.cc:407] Connection header received
0927 12:50:40.434363 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:40.434366 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:40.434408 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:40.434492 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:40.434896 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:40.435646 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:40.436486 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:40.436860 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:40.437281 (+ 421us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:40.441441 (+ 4160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:40.441444 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:40.441446 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:50:40.441467 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:40.443326 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:40.443704 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:40.443707 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:40.443709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:40.443787 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:40.444011 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:40.444014 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:40.444016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:40.444189 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:40.444271 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:40.444496 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 12:50:40.444664 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":244,"thread_start_us":120,"threads_started":1}
I20260927 12:50:40.445669 22802 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35981 } has no permanent_uuid. Determining permanent_uuid...
I20260927 12:50:40.446470 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:40.446204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35981 (local address 127.0.0.1:56440)
0927 12:50:40.446329 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:40.446407 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260927 12:50:40.446688 22750 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35981: connect: Connection refused (error 111)
I20260927 12:50:40.447582 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:40.447361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35981 (local address 127.0.0.1:56448)
0927 12:50:40.447469 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:40.447520 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/krb5kdc/kudu_127.0.98.60.keytab.
W20260927 12:50:40.447753 22802 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35981: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35981: connect: Connection refused (error 111)
I20260927 12:50:40.448767 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35981
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42185,127.0.98.61:38455,127.0.98.60:35981
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 12:50:40.469758 22727 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35981 } attempt: 1
I20260927 12:50:40.470638 22730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:40.470330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35981 (local address 127.0.0.1:56456)
0927 12:50:40.470490 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:40.470555 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 12:50:40.471354 22730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:40.471175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35981 (local address 127.0.0.1:56470)
0927 12:50:40.471260 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:40.471304 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 12:50:40.471472 22727 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35981: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35981: connect: Connection refused (error 111)
I20260927 12:50:40.497033 22802 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35981 } attempt: 1
I20260927 12:50:40.497953 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:40.497591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35981 (local address 127.0.0.1:56482)
0927 12:50:40.497793 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:40.497866 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260927 12:50:40.498792 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:40.498541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35981 (local address 127.0.0.1:56494)
0927 12:50:40.498676 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:40.498729 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 12:50:40.498939 22802 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35981: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35981: connect: Connection refused (error 111)
I20260927 12:50:40.546638 22727 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35981 } attempt: 2
I20260927 12:50:40.547523 22730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:40.547214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35981 (local address 127.0.0.1:56496)
0927 12:50:40.547359 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:40.547422 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 12:50:40.548337 22730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:40.548092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35981 (local address 127.0.0.1:56502)
0927 12:50:40.548200 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:40.548247 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 12:50:40.548488 22727 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35981: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35981: connect: Connection refused (error 111)
W20260927 12:50:40.560428 22810 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 12:50:40.560748 22810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:50:40.560792 22810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:50:40.560858 22810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:50:40.565023 22810 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:50:40.565115 22810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:50:40.565141 22810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:50:40.565164 22810 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:50:40.565196 22810 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:50:40.567143 22802 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35981 } attempt: 2
I20260927 12:50:40.568037 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:40.567719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35981 (local address 127.0.0.1:56506)
0927 12:50:40.567877 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:40.567953 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260927 12:50:40.568857 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:40.568602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35981 (local address 127.0.0.1:56512)
0927 12:50:40.568735 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:40.568789 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260927 12:50:40.568966 22802 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35981: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35981: connect: Connection refused (error 111)
I20260927 12:50:40.571144 22810 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42185,127.0.98.61:38455,127.0.98.60:35981
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:50:40.572288 22810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:50:40.573375 22810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:50:40.581892 22816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:50:40.581895 22815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:50:40.581962 22818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:50:40.582342 22810 server_base.cc:1034] running on GCE node
I20260927 12:50:40.582928 22810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:50:40.584029 22810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:50:40.585201 22810 hybrid_clock.cc:648] HybridClock initialized: now 1790513440585169 us; error 48 us; skew 500 ppm
Sep 27 12:50:40 dist-test-slave-lkn4 krb5kdc[22646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:50:40.588050 22810 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 12:50:40.589105 22810 webserver.cc:460] Webserver started at http://127.0.98.60:34243/ using document root <none> and password file <none>
I20260927 12:50:40.589669 22810 fs_manager.cc:359] Metadata directory not provided
I20260927 12:50:40.589731 22810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:50:40.589902 22810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:50:40.592024 22810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/minicluster-data/master-2/data/instance:
uuid: "c56e23fc238f408d9324731f22002a35"
format_stamp: "Formatted at 2026-09-27 12:50:40 on dist-test-slave-lkn4"
server_key: "265f1b2459c6bfd1084e45943b7d4bbd"
server_key_iv: "560b8ef0cc813d88e7679d20f3d29c70"
server_key_version: "encryptionkey@0"
I20260927 12:50:40.592559 22810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/minicluster-data/master-2/wal/instance:
uuid: "c56e23fc238f408d9324731f22002a35"
format_stamp: "Formatted at 2026-09-27 12:50:40 on dist-test-slave-lkn4"
server_key: "265f1b2459c6bfd1084e45943b7d4bbd"
server_key_iv: "560b8ef0cc813d88e7679d20f3d29c70"
server_key_version: "encryptionkey@0"
I20260927 12:50:40.596401 22810 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260927 12:50:40.599426 22825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:40.600776 22810 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 12:50:40.600920 22810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/minicluster-data/master-2/wal
uuid: "c56e23fc238f408d9324731f22002a35"
format_stamp: "Formatted at 2026-09-27 12:50:40 on dist-test-slave-lkn4"
server_key: "265f1b2459c6bfd1084e45943b7d4bbd"
server_key_iv: "560b8ef0cc813d88e7679d20f3d29c70"
server_key_version: "encryptionkey@0"
I20260927 12:50:40.601029 22810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:50:40.619064 22810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:50:40.622272 22810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:50:40.622476 22810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:50:40.629422 22810 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:35981
I20260927 12:50:40.629673 22877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:35981 every 8 connection(s)
I20260927 12:50:40.630465 22810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/minicluster-data/master-2/data/info.pb
I20260927 12:50:40.633291 22878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:50:40.635149 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 22810
I20260927 12:50:40.635281 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790513161160093-392-0/minicluster-data/master-2/wal/instance
I20260927 12:50:40.635572 392 external_mini_cluster.cc:1428] Setting key 0c75310e73ec95fb22646fbe11576197
I20260927 12:50:40.637655 22878 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42185 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 12:50:40 dist-test-slave-lkn4 krb5kdc[22646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 12:50:40 dist-test-slave-lkn4 krb5kdc[22646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:50:40.647652 22727 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35981 } attempt: 3
I20260927 12:50:40.650496 22729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:40.640049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41956 (local address 127.0.98.62:42185)
0927 12:50:40.640181 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:40.640186 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:40.640465 (+ 279us) server_negotiation.cc:407] Connection header received
0927 12:50:40.640595 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:40.640597 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:40.640639 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:40.640768 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:40.641536 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:40.642105 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:40.643138 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:40.643454 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:40.643534 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:40.646402 (+ 2868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:40.646406 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:40.646409 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:50:40.646439 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:40.648134 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:40.649007 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:40.649011 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:40.649013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:40.649100 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:40.649372 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:40.649375 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:40.649377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:40.649616 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:40.649707 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:40.650204 (+ 497us) server_negotiation.cc:299] Negotiation successful
0927 12:50:40.650337 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 12:50:40.651508 22881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:40.636926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56520 (local address 127.0.98.60:35981)
0927 12:50:40.637323 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:40.637338 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:40.637376 (+ 38us) server_negotiation.cc:407] Connection header received
0927 12:50:40.638077 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:40.638096 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:40.638466 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:40.638760 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:40.639645 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:40.640963 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:40.641942 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:40.642568 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:40.642685 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:40.645906 (+ 3221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:40.645912 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:40.645925 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:50:40.645950 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:40.648023 (+ 2073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:40.648588 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:40.648595 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:40.648596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:40.648754 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:40.649072 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:40.649075 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:40.649076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:40.649596 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:40.649778 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:40.650124 (+ 346us) server_negotiation.cc:299] Negotiation successful
0927 12:50:40.650466 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":258,"thread_start_us":109,"threads_started":1}
I20260927 12:50:40.651733 22882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:40.639948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42185 (local address 127.0.0.1:41956)
0927 12:50:40.640351 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:40.640389 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 12:50:40.640501 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:50:40.640830 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:50:40.640836 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:50:40.641259 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:50:40.641409 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:40.641417 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:40.642214 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:40.642217 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:40.643017 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:40.643027 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:40.643610 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:40.643615 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:40.643734 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:40.644359 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:50:40.644381 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:50:40.646216 (+ 1835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:50:40.648555 (+ 2339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:50:40.648565 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:50:40.648578 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:50:40.648920 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:50:40.649195 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:50:40.649198 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:50:40.649200 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:50:40.649275 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:50:40.649758 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:50:40.649765 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:50:40.649936 (+ 171us) client_negotiation.cc:769] Sending connection context
0927 12:50:40.650638 (+ 702us) client_negotiation.cc:240] Negotiation successful
0927 12:50:40.650836 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":24,"thread_start_us":141,"threads_started":1}
I20260927 12:50:40.652837 22878 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38455 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 12:50:40 dist-test-slave-lkn4 krb5kdc[22646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 12:50:40.657568 22730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:40.648106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35981 (local address 127.0.0.1:56524)
0927 12:50:40.648233 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:40.648247 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:50:40.648345 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:50:40.648900 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:50:40.648903 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:50:40.649062 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:50:40.649171 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:40.649177 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:40.650505 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:40.650509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:40.651277 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:40.651289 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:40.651857 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:40.651862 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:40.651947 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:40.652875 (+ 928us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:50:40.652889 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:50:40.654292 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:50:40.656164 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:50:40.656167 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:50:40.656170 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:50:40.656361 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:50:40.656602 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:50:40.656606 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:50:40.656608 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:50:40.656684 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:50:40.657071 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:50:40.657074 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:50:40.657163 (+ 89us) client_negotiation.cc:769] Sending connection context
0927 12:50:40.657246 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 12:50:40.657400 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 12:50:40.657584 22883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:40.648312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56524 (local address 127.0.98.60:35981)
0927 12:50:40.648645 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:40.648651 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:40.648661 (+ 10us) server_negotiation.cc:407] Connection header received
0927 12:50:40.648689 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:40.648692 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:40.648728 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:40.648801 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:40.649253 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:40.650400 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:40.651400 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:40.651720 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:40.652156 (+ 436us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:40.654432 (+ 2276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:40.654436 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:40.654439 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:50:40.654461 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:40.656096 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:40.656443 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:40.656446 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:40.656448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:40.656526 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:40.656781 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:40.656785 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:40.656787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:40.656981 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:40.657066 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:40.657315 (+ 249us) server_negotiation.cc:299] Negotiation successful
0927 12:50:40.657419 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":272,"thread_start_us":136,"threads_started":1}
Sep 27 12:50:40 dist-test-slave-lkn4 krb5kdc[22646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 27 12:50:40 dist-test-slave-lkn4 krb5kdc[22646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:50:40.660122 22802 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35981 } attempt: 3
I20260927 12:50:40.661135 22727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 976518232da04ad4ab0a261ca6ca25d1: Bootstrap starting.
I20260927 12:50:40.661840 22808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:40.653452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37228 (local address 127.0.98.61:38455)
0927 12:50:40.653574 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:40.653578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:40.653592 (+ 14us) server_negotiation.cc:407] Connection header received
0927 12:50:40.653637 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:40.653640 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:40.653689 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:40.653765 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:40.654222 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:40.654966 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:40.655845 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:40.656110 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:40.656203 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:40.658494 (+ 2291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:40.658498 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:40.658501 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:50:40.658532 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:40.660045 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:40.660550 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:40.660554 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:40.660556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:40.660638 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:40.660927 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:40.660930 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:40.660932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:40.661144 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:40.661212 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:40.661566 (+ 354us) server_negotiation.cc:299] Negotiation successful
0927 12:50:40.661679 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 12:50:40.662278 22882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:40.653327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38455 (local address 127.0.0.1:37228)
0927 12:50:40.653443 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:40.653457 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:50:40.653527 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:50:40.653781 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:50:40.653784 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:50:40.653948 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:50:40.654091 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:40.654099 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:40.655082 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:40.655084 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:40.655733 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:40.655742 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:40.656320 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:40.656324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:40.656420 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:40.656925 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:50:40.656939 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:50:40.658332 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:50:40.660190 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:50:40.660193 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:50:40.660195 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:50:40.660436 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:50:40.660738 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:50:40.660741 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:50:40.660743 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:50:40.660829 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:50:40.661280 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:50:40.661282 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:50:40.661383 (+ 101us) client_negotiation.cc:769] Sending connection context
0927 12:50:40.661976 (+ 593us) client_negotiation.cc:240] Negotiation successful
0927 12:50:40.662131 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260927 12:50:40.662927 22878 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35981 } has no permanent_uuid. Determining permanent_uuid...
I20260927 12:50:40.664036 22729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:40.654345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41966 (local address 127.0.98.62:42185)
0927 12:50:40.654446 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:40.654450 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:40.654462 (+ 12us) server_negotiation.cc:407] Connection header received
0927 12:50:40.654494 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:40.654497 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:40.654532 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:40.654610 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:40.655300 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:40.656015 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:40.657027 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:40.657358 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:40.657456 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:40.660222 (+ 2766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:40.660225 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:40.660228 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:50:40.660259 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:40.662081 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:40.662735 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:40.662738 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:40.662740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:40.662835 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:40.663116 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:40.663119 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:40.663121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:40.663285 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:40.663449 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:40.663631 (+ 182us) server_negotiation.cc:299] Negotiation successful
0927 12:50:40.663932 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 12:50:40.664426 22727 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 976518232da04ad4ab0a261ca6ca25d1: Neither blocks nor log segments found. Creating new log.
I20260927 12:50:40.665301 22727 log.cc:824] T 00000000000000000000000000000000 P 976518232da04ad4ab0a261ca6ca25d1: Log is configured to *not* fsync() on all Append() calls
I20260927 12:50:40.667551 22727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 976518232da04ad4ab0a261ca6ca25d1: No bootstrap required, opened a new log
Sep 27 12:50:40 dist-test-slave-lkn4 krb5kdc[22646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 12:50:40 dist-test-slave-lkn4 krb5kdc[22646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 12:50:40 dist-test-slave-lkn4 krb5kdc[22646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 12:50:40.671949 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:40.660706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35981 (local address 127.0.0.1:56528)
0927 12:50:40.660833 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:40.660847 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:50:40.660912 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:50:40.663722 (+ 2810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:50:40.663726 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:50:40.663898 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:50:40.663996 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:40.664002 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:40.665075 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:40.665079 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:40.665812 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:40.665823 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:40.666413 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:40.666417 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:40.666499 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:40.666981 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:50:40.666995 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:50:40.668432 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:50:40.670334 (+ 1902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:50:40.670337 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:50:40.670340 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:50:40.670600 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:50:40.670920 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:50:40.670923 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:50:40.670925 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:50:40.671014 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:50:40.671452 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:50:40.671456 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:50:40.671551 (+ 95us) client_negotiation.cc:769] Sending connection context
0927 12:50:40.671637 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 12:50:40.671774 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260927 12:50:40.671834 22727 raft_consensus.cc:348] T 00000000000000000000000000000000 P 976518232da04ad4ab0a261ca6ca25d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "976518232da04ad4ab0a261ca6ca25d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42185 } } peers { permanent_uuid: "5e8e89de53064882b7c16ebd1791897d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38455 } } peers { permanent_uuid: "c56e23fc238f408d9324731f22002a35" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35981 } }
I20260927 12:50:40.672106 22727 raft_consensus.cc:729] T 00000000000000000000000000000000 P 976518232da04ad4ab0a261ca6ca25d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 976518232da04ad4ab0a261ca6ca25d1, State: Initialized, Role: FOLLOWER
I20260927 12:50:40.672508 22727 consensus_queue.cc:260] T 00000000000000000000000000000000 P 976518232da04ad4ab0a261ca6ca25d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "976518232da04ad4ab0a261ca6ca25d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42185 } } peers { permanent_uuid: "5e8e89de53064882b7c16ebd1791897d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38455 } } peers { permanent_uuid: "c56e23fc238f408d9324731f22002a35" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35981 } }
I20260927 12:50:40.672676 22881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:40.662708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56528 (local address 127.0.98.60:35981)
0927 12:50:40.663489 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:40.663493 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:40.663505 (+ 12us) server_negotiation.cc:407] Connection header received
0927 12:50:40.663536 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:40.663539 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:40.663576 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:40.664099 (+ 523us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:40.664179 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:40.664957 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:40.665936 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:40.666293 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:40.667569 (+ 1276us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:40.668548 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:40.668551 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:40.668554 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:50:40.668572 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:40.670202 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:40.670721 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:40.670724 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:40.670726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:40.670802 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:40.671119 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:40.671122 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:40.671123 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:40.671344 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:40.672310 (+ 966us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:40.672407 (+ 97us) server_negotiation.cc:299] Negotiation successful
0927 12:50:40.672520 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":724}
I20260927 12:50:40.674158 22727 sys_catalog.cc:564] T 00000000000000000000000000000000 P 976518232da04ad4ab0a261ca6ca25d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:50:40.675021 22882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:40.663368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35981 (local address 127.0.0.1:56530)
0927 12:50:40.665807 (+ 2439us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:40.665819 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:50:40.665892 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:50:40.666175 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:50:40.666178 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:50:40.666347 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:50:40.666422 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:40.666427 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:40.667336 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:40.667339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:40.667937 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:40.667945 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:40.668522 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:40.668526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:40.668596 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:40.669150 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:50:40.669164 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:50:40.671442 (+ 2278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:50:40.673558 (+ 2116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:50:40.673560 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:50:40.673562 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:50:40.673772 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:50:40.674119 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:50:40.674122 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:50:40.674123 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:50:40.674199 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:50:40.674568 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:50:40.674571 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:50:40.674650 (+ 79us) client_negotiation.cc:769] Sending connection context
0927 12:50:40.674740 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 12:50:40.674877 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2398}
I20260927 12:50:40.675427 22883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:40.663403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56530 (local address 127.0.98.60:35981)
0927 12:50:40.665748 (+ 2345us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:40.665752 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:40.665970 (+ 218us) server_negotiation.cc:407] Connection header received
0927 12:50:40.665991 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:40.665994 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:40.666024 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:40.666102 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:40.666515 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:40.667215 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:40.668036 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:40.668338 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:40.668429 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:40.671553 (+ 3124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:40.671556 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:40.671558 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:50:40.671580 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:40.673459 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:40.673878 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:40.673881 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:40.673882 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:40.673953 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:40.674273 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:40.674275 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:40.674277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:40.674450 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:40.674528 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:40.675199 (+ 671us) server_negotiation.cc:299] Negotiation successful
0927 12:50:40.675304 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2276}
I20260927 12:50:40.675923 22802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e8e89de53064882b7c16ebd1791897d: Bootstrap starting.
I20260927 12:50:40.676086 22808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:40.665051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37240 (local address 127.0.98.61:38455)
0927 12:50:40.665146 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:40.665149 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:40.665197 (+ 48us) server_negotiation.cc:407] Connection header received
0927 12:50:40.665323 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:40.665326 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:40.665363 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:40.665422 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:40.666210 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:40.666866 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:40.668529 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:40.668816 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:40.668886 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:40.671710 (+ 2824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:40.671714 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:40.671716 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:50:40.671741 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:40.673645 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:40.674193 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:40.674196 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:40.674198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:40.674279 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:40.674690 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:40.674694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:40.674696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:40.674891 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:40.675063 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:40.675820 (+ 757us) server_negotiation.cc:299] Negotiation successful
0927 12:50:40.675942 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260927 12:50:40.677983 22680 tablet.cc:2369] T 00000000000000000000000000000000 P 976518232da04ad4ab0a261ca6ca25d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:50:40.678297 22885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 976518232da04ad4ab0a261ca6ca25d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "976518232da04ad4ab0a261ca6ca25d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42185 } } peers { permanent_uuid: "5e8e89de53064882b7c16ebd1791897d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38455 } } peers { permanent_uuid: "c56e23fc238f408d9324731f22002a35" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35981 } } }
I20260927 12:50:40.678520 22885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 976518232da04ad4ab0a261ca6ca25d1 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:50:40.679821 22802 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e8e89de53064882b7c16ebd1791897d: Neither blocks nor log segments found. Creating new log.
I20260927 12:50:40.680683 22802 log.cc:824] T 00000000000000000000000000000000 P 5e8e89de53064882b7c16ebd1791897d: Log is configured to *not* fsync() on all Append() calls
I20260927 12:50:40.680730 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:50:40.681371 22878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c56e23fc238f408d9324731f22002a35: Bootstrap starting.
I20260927 12:50:40.682320 22701 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "976518232da04ad4ab0a261ca6ca25d1"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41966
I20260927 12:50:40.682565 22701 raft_consensus.cc:482] T 00000000000000000000000000000000 P 976518232da04ad4ab0a261ca6ca25d1 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 12:50:40.682684 22701 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 976518232da04ad4ab0a261ca6ca25d1 [term 0 FOLLOWER]: Advancing to term 1
W20260927 12:50:40.683419 22896 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 976518232da04ad4ab0a261ca6ca25d1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 12:50:40.683548 22896 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 12:50:40.683818 22878 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c56e23fc238f408d9324731f22002a35: Neither blocks nor log segments found. Creating new log.
I20260927 12:50:40.684031 22701 raft_consensus.cc:504] T 00000000000000000000000000000000 P 976518232da04ad4ab0a261ca6ca25d1 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "976518232da04ad4ab0a261ca6ca25d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42185 } } peers { permanent_uuid: "5e8e89de53064882b7c16ebd1791897d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38455 } } peers { permanent_uuid: "c56e23fc238f408d9324731f22002a35" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35981 } }
I20260927 12:50:40.680827 22659 master_runner.cc:186] Error getting master registration for 127.0.98.61:38455: OK, instance_id { permanent_uuid: "5e8e89de53064882b7c16ebd1791897d" instance_seqno: 1790513440406141 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 12:50:40.684214 22659 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 12:50:40.684548 22878 log.cc:824] T 00000000000000000000000000000000 P c56e23fc238f408d9324731f22002a35: Log is configured to *not* fsync() on all Append() calls
I20260927 12:50:40.686527 22878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c56e23fc238f408d9324731f22002a35: No bootstrap required, opened a new log
I20260927 12:50:40.687126 22701 leader_election.cc:288] T 00000000000000000000000000000000 P 976518232da04ad4ab0a261ca6ca25d1 [CANDIDATE]: Term 1 election: Requested vote from peers 5e8e89de53064882b7c16ebd1791897d (127.0.98.61:38455), c56e23fc238f408d9324731f22002a35 (127.0.98.60:35981)
I20260927 12:50:40.689428 22802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e8e89de53064882b7c16ebd1791897d: No bootstrap required, opened a new log
I20260927 12:50:40.690637 22878 raft_consensus.cc:348] T 00000000000000000000000000000000 P c56e23fc238f408d9324731f22002a35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "976518232da04ad4ab0a261ca6ca25d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42185 } } peers { permanent_uuid: "5e8e89de53064882b7c16ebd1791897d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38455 } } peers { permanent_uuid: "c56e23fc238f408d9324731f22002a35" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35981 } }
I20260927 12:50:40.690930 22878 raft_consensus.cc:729] T 00000000000000000000000000000000 P c56e23fc238f408d9324731f22002a35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c56e23fc238f408d9324731f22002a35, State: Initialized, Role: FOLLOWER
I20260927 12:50:40.691273 22878 consensus_queue.cc:260] T 00000000000000000000000000000000 P c56e23fc238f408d9324731f22002a35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "976518232da04ad4ab0a261ca6ca25d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42185 } } peers { permanent_uuid: "5e8e89de53064882b7c16ebd1791897d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38455 } } peers { permanent_uuid: "c56e23fc238f408d9324731f22002a35" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35981 } }
I20260927 12:50:40.692224 22900 sys_catalog.cc:455] T 00000000000000000000000000000000 P c56e23fc238f408d9324731f22002a35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "976518232da04ad4ab0a261ca6ca25d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42185 } } peers { permanent_uuid: "5e8e89de53064882b7c16ebd1791897d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38455 } } peers { permanent_uuid: "c56e23fc238f408d9324731f22002a35" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35981 } } }
I20260927 12:50:40.692713 22878 sys_catalog.cc:564] T 00000000000000000000000000000000 P c56e23fc238f408d9324731f22002a35 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:50:40.693534 22802 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e8e89de53064882b7c16ebd1791897d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "976518232da04ad4ab0a261ca6ca25d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42185 } } peers { permanent_uuid: "5e8e89de53064882b7c16ebd1791897d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38455 } } peers { permanent_uuid: "c56e23fc238f408d9324731f22002a35" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35981 } }
I20260927 12:50:40.693782 22802 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e8e89de53064882b7c16ebd1791897d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e8e89de53064882b7c16ebd1791897d, State: Initialized, Role: FOLLOWER
I20260927 12:50:40.693498 22852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "976518232da04ad4ab0a261ca6ca25d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c56e23fc238f408d9324731f22002a35"
I20260927 12:50:40.694190 22852 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c56e23fc238f408d9324731f22002a35 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:50:40.694150 22802 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e8e89de53064882b7c16ebd1791897d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "976518232da04ad4ab0a261ca6ca25d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42185 } } peers { permanent_uuid: "5e8e89de53064882b7c16ebd1791897d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38455 } } peers { permanent_uuid: "c56e23fc238f408d9324731f22002a35" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35981 } }
I20260927 12:50:40.695514 22802 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e8e89de53064882b7c16ebd1791897d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:50:40.695690 22852 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c56e23fc238f408d9324731f22002a35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 976518232da04ad4ab0a261ca6ca25d1 in term 1.
I20260927 12:50:40.696274 22675 leader_election.cc:302] T 00000000000000000000000000000000 P 976518232da04ad4ab0a261ca6ca25d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 976518232da04ad4ab0a261ca6ca25d1, c56e23fc238f408d9324731f22002a35; no voters:
I20260927 12:50:40.696643 22885 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 976518232da04ad4ab0a261ca6ca25d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:50:40.696861 22885 raft_consensus.cc:686] T 00000000000000000000000000000000 P 976518232da04ad4ab0a261ca6ca25d1 [term 1 LEADER]: Becoming Leader. State: Replica: 976518232da04ad4ab0a261ca6ca25d1, State: Running, Role: LEADER
I20260927 12:50:40.697247 22885 consensus_queue.cc:237] T 00000000000000000000000000000000 P 976518232da04ad4ab0a261ca6ca25d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "976518232da04ad4ab0a261ca6ca25d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42185 } } peers { permanent_uuid: "5e8e89de53064882b7c16ebd1791897d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38455 } } peers { permanent_uuid: "c56e23fc238f408d9324731f22002a35" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35981 } }
I20260927 12:50:40.699277 22885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 976518232da04ad4ab0a261ca6ca25d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 976518232da04ad4ab0a261ca6ca25d1. Latest consensus state: current_term: 1 leader_uuid: "976518232da04ad4ab0a261ca6ca25d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "976518232da04ad4ab0a261ca6ca25d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42185 } } peers { permanent_uuid: "5e8e89de53064882b7c16ebd1791897d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38455 } } peers { permanent_uuid: "c56e23fc238f408d9324731f22002a35" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35981 } } }
I20260927 12:50:40.699417 22885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 976518232da04ad4ab0a261ca6ca25d1 [sys.catalog]: This master's current role is: LEADER
W20260927 12:50:40.701011 22930 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5e8e89de53064882b7c16ebd1791897d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 12:50:40.701081 22930 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 12:50:40.701679 22933 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:50:40.701589 22776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "976518232da04ad4ab0a261ca6ca25d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5e8e89de53064882b7c16ebd1791897d"
I20260927 12:50:40.701839 22776 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e8e89de53064882b7c16ebd1791897d [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:50:40.702723 22933 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:50:40.703289 22776 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5e8e89de53064882b7c16ebd1791897d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 976518232da04ad4ab0a261ca6ca25d1 in term 1.
I20260927 12:50:40.707811 22729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:40.696675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41980 (local address 127.0.98.62:42185)
0927 12:50:40.696800 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:40.696806 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:40.698310 (+ 1504us) server_negotiation.cc:407] Connection header received
0927 12:50:40.698449 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:40.698452 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:40.698502 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:40.698589 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:40.699578 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:40.700314 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:40.701424 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:40.701734 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:40.701846 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:40.703715 (+ 1869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:40.703719 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:40.703721 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:50:40.703749 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:40.705437 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:40.705956 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:40.705960 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:40.705962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:40.706077 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:40.706387 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:40.706390 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:40.706392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:40.706620 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:40.706742 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:40.707520 (+ 778us) server_negotiation.cc:299] Negotiation successful
0927 12:50:40.707640 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 12:50:40.708896 22900 sys_catalog.cc:458] T 00000000000000000000000000000000 P c56e23fc238f408d9324731f22002a35 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:50:40.708889 22908 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e8e89de53064882b7c16ebd1791897d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "976518232da04ad4ab0a261ca6ca25d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42185 } } peers { permanent_uuid: "5e8e89de53064882b7c16ebd1791897d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38455 } } peers { permanent_uuid: "c56e23fc238f408d9324731f22002a35" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35981 } } }
I20260927 12:50:40.709013 22908 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e8e89de53064882b7c16ebd1791897d [sys.catalog]: This master's current role is: FOLLOWER
W20260927 12:50:40.712975 22918 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c56e23fc238f408d9324731f22002a35: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260927 12:50:40.722210 22808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:40.709665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37250 (local address 127.0.98.61:38455)
0927 12:50:40.709781 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:40.709786 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:40.713482 (+ 3696us) server_negotiation.cc:407] Connection header received
0927 12:50:40.713689 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:40.713692 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:40.713747 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:40.713806 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:40.714735 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:40.715468 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:40.716584 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:40.716904 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:40.716987 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:40.718790 (+ 1803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:40.718794 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:40.718797 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:50:40.718823 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:40.720403 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:40.720877 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:40.720881 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:40.720882 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:40.720965 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:40.721246 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:40.721249 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:40.721251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:40.721457 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:40.721560 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:40.721914 (+ 354us) server_negotiation.cc:299] Negotiation successful
0927 12:50:40.722026 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 12:50:40.717012 22881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:40.697537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56540 (local address 127.0.98.60:35981)
0927 12:50:40.697713 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:40.697717 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:40.707422 (+ 9705us) server_negotiation.cc:407] Connection header received
0927 12:50:40.708119 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:40.708122 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:40.708168 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:40.708215 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:40.709127 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:40.709974 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:40.711123 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:40.711453 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:40.711530 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:40.713300 (+ 1770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:40.713305 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:40.713308 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:50:40.713337 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:40.715145 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:40.715633 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:40.715637 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:40.715639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:40.715722 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:40.716031 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:40.716034 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:40.716036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:40.716251 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:40.716351 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:40.716712 (+ 361us) server_negotiation.cc:299] Negotiation successful
0927 12:50:40.716846 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
W20260927 12:50:40.723995 22918 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 12:50:40.730530 22852 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c56e23fc238f408d9324731f22002a35 [term 1 FOLLOWER]: Refusing update from remote peer 976518232da04ad4ab0a261ca6ca25d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:50:40.730574 22776 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5e8e89de53064882b7c16ebd1791897d [term 1 FOLLOWER]: Refusing update from remote peer 976518232da04ad4ab0a261ca6ca25d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:50:40.733481 22885 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 976518232da04ad4ab0a261ca6ca25d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e8e89de53064882b7c16ebd1791897d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:50:40.738819 22908 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e8e89de53064882b7c16ebd1791897d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 976518232da04ad4ab0a261ca6ca25d1. Latest consensus state: current_term: 1 leader_uuid: "976518232da04ad4ab0a261ca6ca25d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "976518232da04ad4ab0a261ca6ca25d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42185 } } peers { permanent_uuid: "5e8e89de53064882b7c16ebd1791897d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38455 } } peers { permanent_uuid: "c56e23fc238f408d9324731f22002a35" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35981 } } }
I20260927 12:50:40.738986 22908 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e8e89de53064882b7c16ebd1791897d [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:50:40.739996 22885 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 976518232da04ad4ab0a261ca6ca25d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c56e23fc238f408d9324731f22002a35" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:50:40.744379 22900 sys_catalog.cc:455] T 00000000000000000000000000000000 P c56e23fc238f408d9324731f22002a35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 976518232da04ad4ab0a261ca6ca25d1. Latest consensus state: current_term: 1 leader_uuid: "976518232da04ad4ab0a261ca6ca25d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "976518232da04ad4ab0a261ca6ca25d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42185 } } peers { permanent_uuid: "5e8e89de53064882b7c16ebd1791897d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38455 } } peers { permanent_uuid: "c56e23fc238f408d9324731f22002a35" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35981 } } }
I20260927 12:50:40.744524 22900 sys_catalog.cc:458] T 00000000000000000000000000000000 P c56e23fc238f408d9324731f22002a35 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:50:40.757050 22885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 976518232da04ad4ab0a261ca6ca25d1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "976518232da04ad4ab0a261ca6ca25d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "976518232da04ad4ab0a261ca6ca25d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42185 } } peers { permanent_uuid: "5e8e89de53064882b7c16ebd1791897d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38455 } } peers { permanent_uuid: "c56e23fc238f408d9324731f22002a35" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35981 } } }
I20260927 12:50:40.757203 22885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 976518232da04ad4ab0a261ca6ca25d1 [sys.catalog]: This master's current role is: LEADER
I20260927 12:50:40.757318 22935 mvcc.cc:205] Tried to move back new op lower bound from 7333943053221343232 to 7333943053099847680. Current Snapshot: MvccSnapshot[applied={T|T < 7333943053221343232}]
I20260927 12:50:40.758595 22900 sys_catalog.cc:455] T 00000000000000000000000000000000 P c56e23fc238f408d9324731f22002a35 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "976518232da04ad4ab0a261ca6ca25d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "976518232da04ad4ab0a261ca6ca25d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42185 } } peers { permanent_uuid: "5e8e89de53064882b7c16ebd1791897d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38455 } } peers { permanent_uuid: "c56e23fc238f408d9324731f22002a35" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35981 } } }
I20260927 12:50:40.758720 22900 sys_catalog.cc:458] T 00000000000000000000000000000000 P c56e23fc238f408d9324731f22002a35 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:50:40.760182 22938 mvcc.cc:205] Tried to move back new op lower bound from 7333943053221343232 to 7333943053099847680. Current Snapshot: MvccSnapshot[applied={T|T < 7333943053221343232}]
I20260927 12:50:40.760941 22923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 976518232da04ad4ab0a261ca6ca25d1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "976518232da04ad4ab0a261ca6ca25d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "976518232da04ad4ab0a261ca6ca25d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42185 } } peers { permanent_uuid: "5e8e89de53064882b7c16ebd1791897d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38455 } } peers { permanent_uuid: "c56e23fc238f408d9324731f22002a35" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35981 } } }
I20260927 12:50:40.761058 22923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 976518232da04ad4ab0a261ca6ca25d1 [sys.catalog]: This master's current role is: LEADER
I20260927 12:50:40.764047 22908 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e8e89de53064882b7c16ebd1791897d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "976518232da04ad4ab0a261ca6ca25d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "976518232da04ad4ab0a261ca6ca25d1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42185 } } peers { permanent_uuid: "5e8e89de53064882b7c16ebd1791897d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38455 } } peers { permanent_uuid: "c56e23fc238f408d9324731f22002a35" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35981 } } }
I20260927 12:50:40.764159 22908 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e8e89de53064882b7c16ebd1791897d [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:50:40.765933 22936 mvcc.cc:205] Tried to move back new op lower bound from 7333943053221343232 to 7333943053099847680. Current Snapshot: MvccSnapshot[applied={T|T < 7333943053221343232}]
I20260927 12:50:40.769603 22933 catalog_manager.cc:1347] Generated new cluster ID: 6908324565564a9599efd9e7d0286577
I20260927 12:50:40.769680 22933 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:50:40.801469 22933 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:50:40.802886 22933 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:50:40.831591 22933 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 976518232da04ad4ab0a261ca6ca25d1: Generated new TSK 0
I20260927 12:50:40.832580 22933 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 12:50:40.875342 22808 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:50:40.870549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37252 (local address 127.0.98.61:38455)
0927 12:50:40.870696 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:40.870702 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:40.872848 (+ 2146us) server_negotiation.cc:407] Connection header received
0927 12:50:40.873148 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:40.873153 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:40.873212 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:40.873282 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:40.875182 (+ 1900us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37252: BlockingRecv error: recv got EOF from 127.0.0.1:37252 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260927 12:50:40.877161 22729 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:50:40.871491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41992 (local address 127.0.98.62:42185)
0927 12:50:40.871611 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:40.871616 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:40.876120 (+ 4504us) server_negotiation.cc:407] Connection header received
0927 12:50:40.876339 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:40.876342 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:40.876400 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:40.876454 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:40.877029 (+ 575us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41992: BlockingRecv error: recv got EOF from 127.0.0.1:41992 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260927 12:50:40.878821 22881 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:50:40.872181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56550 (local address 127.0.98.60:35981)
0927 12:50:40.872322 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:40.872327 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:40.877836 (+ 5509us) server_negotiation.cc:407] Connection header received
0927 12:50:40.878020 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:40.878024 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:40.878120 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:40.878175 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:40.878697 (+ 522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56550: BlockingRecv error: recv got EOF from 127.0.0.1:56550 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260927 12:50:40.884044 22808 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:50:40.882273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37266 (local address 127.0.98.61:38455)
0927 12:50:40.882387 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:40.882392 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:40.882952 (+ 560us) server_negotiation.cc:407] Connection header received
0927 12:50:40.883191 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:40.883195 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:40.883292 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:40.883352 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:40.883926 (+ 574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37266: BlockingRecv error: recv got EOF from 127.0.0.1:37266 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260927 12:50:40.889994 22808 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:50:40.887962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37280 (local address 127.0.98.61:38455)
0927 12:50:40.888075 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:40.888080 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:40.888700 (+ 620us) server_negotiation.cc:407] Connection header received
0927 12:50:40.889169 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:40.889173 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:40.889231 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:40.889303 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:40.889876 (+ 573us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37280: BlockingRecv error: recv got EOF from 127.0.0.1:37280 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260927 12:50:40.896626 22808 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:50:40.894966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37284 (local address 127.0.98.61:38455)
0927 12:50:40.895077 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:40.895082 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:40.895671 (+ 589us) server_negotiation.cc:407] Connection header received
0927 12:50:40.895865 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:40.895869 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:40.895919 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:40.895968 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:40.896505 (+ 537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37284: BlockingRecv error: recv got EOF from 127.0.0.1:37284 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260927 12:50:40.904983 22808 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:50:40.903198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37290 (local address 127.0.98.61:38455)
0927 12:50:40.903349 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:40.903354 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:40.903884 (+ 530us) server_negotiation.cc:407] Connection header received
0927 12:50:40.904089 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:40.904093 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:40.904163 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:40.904216 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:40.904864 (+ 648us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37290: BlockingRecv error: recv got EOF from 127.0.0.1:37290 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260927 12:50:40.917220 22808 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:50:40.915863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37302 (local address 127.0.98.61:38455)
0927 12:50:40.915983 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:40.915988 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:40.916282 (+ 294us) server_negotiation.cc:407] Connection header received
0927 12:50:40.916407 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:40.916410 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:40.916468 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:40.916552 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:40.917050 (+ 498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37302: BlockingRecv error: recv got EOF from 127.0.0.1:37302 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260927 12:50:40.936339 22808 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:50:40.935168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37318 (local address 127.0.98.61:38455)
0927 12:50:40.935330 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:40.935335 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:40.935443 (+ 108us) server_negotiation.cc:407] Connection header received
0927 12:50:40.935596 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:40.935600 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:40.935648 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:40.935732 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:40.936219 (+ 487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37318: BlockingRecv error: recv got EOF from 127.0.0.1:37318 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260927 12:50:40.971478 22808 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:50:40.970389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37326 (local address 127.0.98.61:38455)
0927 12:50:40.970544 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:40.970548 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:40.970632 (+ 84us) server_negotiation.cc:407] Connection header received
0927 12:50:40.970764 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:40.970804 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:40.970859 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:40.970949 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:40.971376 (+ 427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37326: BlockingRecv error: recv got EOF from 127.0.0.1:37326 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260927 12:50:41.038717 22808 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:50:41.037592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37328 (local address 127.0.98.61:38455)
0927 12:50:41.037728 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:41.037733 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:41.037923 (+ 190us) server_negotiation.cc:407] Connection header received
0927 12:50:41.038070 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:41.038073 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:41.038125 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:41.038191 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:41.038595 (+ 404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37328: BlockingRecv error: recv got EOF from 127.0.0.1:37328 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260927 12:50:41.170185 22808 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:50:41.168937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37342 (local address 127.0.98.61:38455)
0927 12:50:41.169078 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:41.169082 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:41.169250 (+ 168us) server_negotiation.cc:407] Connection header received
0927 12:50:41.169414 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:41.169417 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:41.169466 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:41.169536 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:41.170062 (+ 526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37342: BlockingRecv error: recv got EOF from 127.0.0.1:37342 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260927 12:50:41.429354 22808 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:50:41.428345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37358 (local address 127.0.98.61:38455)
0927 12:50:41.428467 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:41.428470 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:41.428530 (+ 60us) server_negotiation.cc:407] Connection header received
0927 12:50:41.428653 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:41.428656 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:41.428702 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:41.428785 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:41.429247 (+ 462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37358: BlockingRecv error: recv got EOF from 127.0.0.1:37358 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 12:50:41.711326 22930 catalog_manager.cc:1259] Loaded cluster ID: 6908324565564a9599efd9e7d0286577
I20260927 12:50:41.711422 22930 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5e8e89de53064882b7c16ebd1791897d: loading cluster ID for follower catalog manager: success
I20260927 12:50:41.713195 22930 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5e8e89de53064882b7c16ebd1791897d: acquiring CA information for follower catalog manager: success
I20260927 12:50:41.714453 22930 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5e8e89de53064882b7c16ebd1791897d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 12:50:41.726848 22918 catalog_manager.cc:1259] Loaded cluster ID: 6908324565564a9599efd9e7d0286577
I20260927 12:50:41.726936 22918 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c56e23fc238f408d9324731f22002a35: loading cluster ID for follower catalog manager: success
I20260927 12:50:41.728605 22918 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c56e23fc238f408d9324731f22002a35: acquiring CA information for follower catalog manager: success
I20260927 12:50:41.729777 22918 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c56e23fc238f408d9324731f22002a35: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 12:50:41.947391 23008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:41.943406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42912 (local address 127.0.98.61:38455)
0927 12:50:41.943632 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:41.943636 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:41.943651 (+ 15us) server_negotiation.cc:407] Connection header received
0927 12:50:41.943759 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:41.943761 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:41.943888 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:41.943998 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:50:41.944348 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:41.944918 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:41.945876 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:41.946155 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:41.946248 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:41.946542 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:50:41.947009 (+ 467us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:50:41.947138 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:41.947194 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 12:50:41.947251 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":64,"threads_started":1}
I20260927 12:50:41.954315 23014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:41.950646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37398 (local address 127.0.98.60:35981)
0927 12:50:41.950894 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:41.950897 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:41.950911 (+ 14us) server_negotiation.cc:407] Connection header received
0927 12:50:41.951037 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:41.951039 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:41.951152 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:41.951246 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:50:41.951494 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:41.952016 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:41.952847 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:41.953075 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:41.953153 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:41.953452 (+ 299us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:50:41.953943 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:50:41.954077 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:41.954112 (+ 35us) server_negotiation.cc:299] Negotiation successful
0927 12:50:41.954163 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":88,"threads_started":1}
I20260927 12:50:41.961413 23008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:41.957399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42928 (local address 127.0.98.61:38455)
0927 12:50:41.957527 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:41.957532 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:41.957716 (+ 184us) server_negotiation.cc:407] Connection header received
0927 12:50:41.957883 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:41.957886 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:41.957942 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:41.957989 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:50:41.958433 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:41.959077 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:41.960353 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:41.960665 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:41.960751 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:41.960990 (+ 239us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:50:41.961060 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:50:41.961115 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:41.961214 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 12:50:41.961269 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 12:50:41.961792 23014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:41.958113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37410 (local address 127.0.98.60:35981)
0927 12:50:41.958254 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:41.958258 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:41.958269 (+ 11us) server_negotiation.cc:407] Connection header received
0927 12:50:41.958433 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:41.958436 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:41.958480 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:41.958523 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:50:41.958861 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:41.959570 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:41.960598 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:41.960895 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:41.960962 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:41.961345 (+ 383us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:50:41.961423 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:50:41.961462 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:41.961617 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 12:50:41.961676 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 12:50:41.961938 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:41.957429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35956 (local address 127.0.98.62:42185)
0927 12:50:41.957745 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:41.957750 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:41.957769 (+ 19us) server_negotiation.cc:407] Connection header received
0927 12:50:41.957918 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:41.957924 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:41.958158 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:41.958291 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:50:41.958535 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:41.959171 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:41.960174 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:41.960453 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:41.960567 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:41.960886 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:50:41.961251 (+ 365us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:50:41.961665 (+ 414us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:41.961696 (+ 31us) server_negotiation.cc:299] Negotiation successful
0927 12:50:41.961752 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":79,"threads_started":1}
I20260927 12:50:41.969810 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 22659
I20260927 12:50:41.975451 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 22734
I20260927 12:50:41.980491 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 22810
2026-09-27T12:50:41Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3053 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:50:42 dist-test-slave-lkn4 krb5kdc[23025](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:50:42 dist-test-slave-lkn4 krb5kdc[23025](info): set up 2 sockets
Sep 27 12:50:42 dist-test-slave-lkn4 krb5kdc[23025](info): commencing operation
krb5kdc: starting...
W20260927 12:50:42.934335 392 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/krb5kdc/test-user.keytab.
Sep 27 12:50:43 dist-test-slave-lkn4 krb5kdc[23025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:50:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:50:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:50:43.055114 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38661
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38661,127.0.98.61:37335,127.0.98.60:41253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:50:43.165757 23038 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 12:50:43.166080 23038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:50:43.166121 23038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:50:43.166165 23038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:50:43.170207 23038 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:50:43.170274 23038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:50:43.170292 23038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:50:43.170308 23038 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:50:43.170323 23038 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:50:43.175738 23038 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38661,127.0.98.61:37335,127.0.98.60:41253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:50:43.176837 23038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:50:43.177860 23038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 12:50:43.186312 23038 server_base.cc:1034] running on GCE node
W20260927 12:50:43.186306 23043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:50:43.186334 23046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:50:43.186295 23044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:50:43.187105 23038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:50:43.188283 23038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:50:43.189464 23038 hybrid_clock.cc:648] HybridClock initialized: now 1790513443189446 us; error 29 us; skew 500 ppm
Sep 27 12:50:43 dist-test-slave-lkn4 krb5kdc[23025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:50:43.192262 23038 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:50:43.193169 23038 webserver.cc:460] Webserver started at http://127.0.98.62:43785/ using document root <none> and password file <none>
I20260927 12:50:43.193701 23038 fs_manager.cc:359] Metadata directory not provided
I20260927 12:50:43.193758 23038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:50:43.193925 23038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:50:43.196096 23038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/minicluster-data/master-0/data/instance:
uuid: "bf0c1b8cfb65477fadd3db86d7af257d"
format_stamp: "Formatted at 2026-09-27 12:50:43 on dist-test-slave-lkn4"
server_key: "31b7e71888f0b6dacdb5e1b3205cc5c5"
server_key_iv: "87e184dfae3bd853605aa751a0d3146d"
server_key_version: "encryptionkey@0"
I20260927 12:50:43.196657 23038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/minicluster-data/master-0/wal/instance:
uuid: "bf0c1b8cfb65477fadd3db86d7af257d"
format_stamp: "Formatted at 2026-09-27 12:50:43 on dist-test-slave-lkn4"
server_key: "31b7e71888f0b6dacdb5e1b3205cc5c5"
server_key_iv: "87e184dfae3bd853605aa751a0d3146d"
server_key_version: "encryptionkey@0"
I20260927 12:50:43.200683 23038 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 12:50:43.203325 23053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:43.204555 23038 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 12:50:43.204656 23038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/minicluster-data/master-0/wal
uuid: "bf0c1b8cfb65477fadd3db86d7af257d"
format_stamp: "Formatted at 2026-09-27 12:50:43 on dist-test-slave-lkn4"
server_key: "31b7e71888f0b6dacdb5e1b3205cc5c5"
server_key_iv: "87e184dfae3bd853605aa751a0d3146d"
server_key_version: "encryptionkey@0"
I20260927 12:50:43.204746 23038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:50:43.223551 23038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:50:43.226791 23038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:50:43.226953 23038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:50:43.233515 23038 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38661
I20260927 12:50:43.233794 23105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38661 every 8 connection(s)
I20260927 12:50:43.234594 23038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:50:43.237465 23106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:50:43.241605 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 23038
I20260927 12:50:43.241372 23106 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38661 } has no permanent_uuid. Determining permanent_uuid...
I20260927 12:50:43.241729 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/minicluster-data/master-0/wal/instance
I20260927 12:50:43.242017 392 external_mini_cluster.cc:1428] Setting key 1b9dcd32a2da9cf0e79fcb990a76efef
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 27 12:50:43 dist-test-slave-lkn4 krb5kdc[23025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:50:43.258841 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:43.243902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38661 (local address 127.0.0.1:46760)
0927 12:50:43.244402 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:43.244457 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 12:50:43.245483 (+ 1026us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:50:43.246299 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:50:43.246305 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:50:43.246781 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:50:43.247133 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:43.247153 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:43.248327 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:43.248330 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:43.249497 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:43.249508 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:43.250220 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:43.250224 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:43.250371 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:43.251001 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:50:43.251019 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:50:43.253817 (+ 2798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:50:43.255925 (+ 2108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:50:43.255934 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:50:43.255945 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:50:43.256169 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:50:43.256488 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:50:43.256491 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:50:43.256493 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:50:43.256596 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:50:43.257191 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:50:43.257197 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:50:43.257365 (+ 168us) client_negotiation.cc:769] Sending connection context
0927 12:50:43.257568 (+ 203us) client_negotiation.cc:240] Negotiation successful
0927 12:50:43.258087 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":102,"thread_start_us":120,"threads_started":1}
I20260927 12:50:43.258842 23110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:43.243903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46760 (local address 127.0.98.62:38661)
0927 12:50:43.244475 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:43.244486 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:43.244587 (+ 101us) server_negotiation.cc:407] Connection header received
0927 12:50:43.245827 (+ 1240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:43.245832 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:43.246035 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:43.246270 (+ 235us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:43.247300 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:43.248190 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:43.249622 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:43.250017 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:43.250185 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:43.254014 (+ 3829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:43.254020 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:43.254032 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:50:43.254094 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:43.255810 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:43.256269 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:43.256276 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:43.256278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:43.256397 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:43.256673 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:43.256677 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:43.256678 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:43.257078 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:43.257192 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:43.257741 (+ 549us) server_negotiation.cc:299] Negotiation successful
0927 12:50:43.257939 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"spinlock_wait_cycles":57088,"thread_start_us":167,"threads_started":1}
I20260927 12:50:43.261478 23106 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37335 } has no permanent_uuid. Determining permanent_uuid...
I20260927 12:50:43.262317 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:43.261951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37335 (local address 127.0.0.1:55950)
0927 12:50:43.262163 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:43.262244 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260927 12:50:43.262537 23057 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37335: connect: Connection refused (error 111)
I20260927 12:50:43.263458 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:43.263225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37335 (local address 127.0.0.1:55960)
0927 12:50:43.263344 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:43.263399 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 12:50:43.263604 23106 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37335: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37335: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 12:50:43.269456 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:37335
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38661,127.0.98.61:37335,127.0.98.60:41253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 12:50:43.312860 23106 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37335 } attempt: 1
I20260927 12:50:43.313751 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:43.313425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37335 (local address 127.0.0.1:55966)
0927 12:50:43.313597 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:43.313670 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 12:50:43.314457 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:43.314225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37335 (local address 127.0.0.1:55982)
0927 12:50:43.314351 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:43.314399 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260927 12:50:43.314602 23106 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37335: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37335: connect: Connection refused (error 111)
W20260927 12:50:43.380118 23113 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 12:50:43.380430 23113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:50:43.380455 23113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:50:43.380505 23113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 12:50:43.382786 23106 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37335 } attempt: 2
I20260927 12:50:43.383540 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:43.383275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37335 (local address 127.0.0.1:55996)
0927 12:50:43.383408 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:43.383465 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 12:50:43.384176 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:43.383982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37335 (local address 127.0.0.1:56004)
0927 12:50:43.384077 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:43.384120 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 12:50:43.384286 23106 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37335: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37335: connect: Connection refused (error 111)
W20260927 12:50:43.384879 23113 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:50:43.384929 23113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:50:43.384948 23113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:50:43.384963 23113 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:50:43.384982 23113 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:50:43.390358 23113 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38661,127.0.98.61:37335,127.0.98.60:41253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:37335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23113
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:50:43.391417 23113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:50:43.392458 23113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:50:43.401180 23118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:50:43.401219 23119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:50:43.401353 23113 server_base.cc:1034] running on GCE node
W20260927 12:50:43.401182 23121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:50:43.402311 23113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:50:43.403452 23113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:50:43.404637 23113 hybrid_clock.cc:648] HybridClock initialized: now 1790513443404620 us; error 33 us; skew 500 ppm
Sep 27 12:50:43 dist-test-slave-lkn4 krb5kdc[23025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:50:43.407471 23113 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 12:50:43.408525 23113 webserver.cc:460] Webserver started at http://127.0.98.61:37479/ using document root <none> and password file <none>
I20260927 12:50:43.409099 23113 fs_manager.cc:359] Metadata directory not provided
I20260927 12:50:43.409173 23113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:50:43.409381 23113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:50:43.411563 23113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/minicluster-data/master-1/data/instance:
uuid: "217eb45bb3e34dd58a4275a4e4bc24d0"
format_stamp: "Formatted at 2026-09-27 12:50:43 on dist-test-slave-lkn4"
server_key: "8ea142e2ceaeae8b4ecb07c3b0e43b7a"
server_key_iv: "f8b1829d012b67686d9c78bc8f09060f"
server_key_version: "encryptionkey@0"
I20260927 12:50:43.412132 23113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/minicluster-data/master-1/wal/instance:
uuid: "217eb45bb3e34dd58a4275a4e4bc24d0"
format_stamp: "Formatted at 2026-09-27 12:50:43 on dist-test-slave-lkn4"
server_key: "8ea142e2ceaeae8b4ecb07c3b0e43b7a"
server_key_iv: "f8b1829d012b67686d9c78bc8f09060f"
server_key_version: "encryptionkey@0"
I20260927 12:50:43.416047 23113 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 12:50:43.418555 23128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:43.419898 23113 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:50:43.420019 23113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/minicluster-data/master-1/wal
uuid: "217eb45bb3e34dd58a4275a4e4bc24d0"
format_stamp: "Formatted at 2026-09-27 12:50:43 on dist-test-slave-lkn4"
server_key: "8ea142e2ceaeae8b4ecb07c3b0e43b7a"
server_key_iv: "f8b1829d012b67686d9c78bc8f09060f"
server_key_version: "encryptionkey@0"
I20260927 12:50:43.420123 23113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:50:43.444000 23113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:50:43.447223 23113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:50:43.447414 23113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:50:43.454386 23113 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:37335
I20260927 12:50:43.454653 23180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:37335 every 8 connection(s)
I20260927 12:50:43.455396 23113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/minicluster-data/master-1/data/info.pb
I20260927 12:50:43.455682 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 23113
I20260927 12:50:43.455859 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/minicluster-data/master-1/wal/instance
I20260927 12:50:43.456161 392 external_mini_cluster.cc:1428] Setting key a48b68c8e48484a164e12de99ace1150
I20260927 12:50:43.458335 23181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:50:43.463295 23181 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38661 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 27 12:50:43 dist-test-slave-lkn4 krb5kdc[23025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:50:43.475503 23106 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37335 } attempt: 3
I20260927 12:50:43.477731 23110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:43.465422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46774 (local address 127.0.98.62:38661)
0927 12:50:43.465563 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:43.465567 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:43.466488 (+ 921us) server_negotiation.cc:407] Connection header received
0927 12:50:43.467654 (+ 1166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:43.467657 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:43.467707 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:43.467782 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:43.469139 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:43.469939 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:43.471183 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:43.471560 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:43.471669 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:43.474237 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:43.474242 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:43.474245 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:50:43.474283 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:43.475685 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:43.476212 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:43.476216 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:43.476218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:43.476317 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:43.476648 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:43.476651 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:43.476653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:43.476849 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:43.476940 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:43.477486 (+ 546us) server_negotiation.cc:299] Negotiation successful
0927 12:50:43.477581 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 12:50:43.478765 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:43.465761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38661 (local address 127.0.0.1:46774)
0927 12:50:43.466330 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:43.466374 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 12:50:43.467417 (+ 1043us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:50:43.467984 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:50:43.467992 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:50:43.468460 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:50:43.469001 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:43.469015 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:43.470091 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:43.470094 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:43.471064 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:43.471073 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:43.471697 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:43.471700 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:43.471890 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:43.472441 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:50:43.472457 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:50:43.474069 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:50:43.475785 (+ 1716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:50:43.475791 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:50:43.475808 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:50:43.476091 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:50:43.476420 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:50:43.476422 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:50:43.476423 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:50:43.476550 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:50:43.476954 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:50:43.476961 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:50:43.477228 (+ 267us) client_negotiation.cc:769] Sending connection context
0927 12:50:43.477424 (+ 196us) client_negotiation.cc:240] Negotiation successful
0927 12:50:43.477753 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":122,"threads_started":1}
I20260927 12:50:43.480592 23181 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37335 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 12:50:43 dist-test-slave-lkn4 krb5kdc[23025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260927 12:50:43.484715 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
/tmp/dist-test-taskTC8Et9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41253
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38661,127.0.98.61:37335,127.0.98.60:41253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 12:50:43 dist-test-slave-lkn4 krb5kdc[23025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 12:50:43.486208 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:43.475957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37335 (local address 127.0.0.1:56010)
0927 12:50:43.476096 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:43.476122 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 12:50:43.476203 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:50:43.476864 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:50:43.476867 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:50:43.477030 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:50:43.477126 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:43.477132 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:43.478496 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:43.478498 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:43.479120 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:43.479130 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:43.479932 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:43.479936 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:43.480038 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:43.480464 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:50:43.480474 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:50:43.482187 (+ 1713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:50:43.484327 (+ 2140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:50:43.484330 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:50:43.484332 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:50:43.484584 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:50:43.484963 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:50:43.484966 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:50:43.484968 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:50:43.485056 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:50:43.485654 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:50:43.485657 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:50:43.485762 (+ 105us) client_negotiation.cc:769] Sending connection context
0927 12:50:43.485851 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 12:50:43.485990 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260927 12:50:43.486881 23186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:43.476069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56010 (local address 127.0.98.61:37335)
0927 12:50:43.476468 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:43.476481 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:43.476507 (+ 26us) server_negotiation.cc:407] Connection header received
0927 12:50:43.476553 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:43.476557 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:43.476737 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:43.476853 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:43.477222 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:43.478404 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:43.479249 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:43.479825 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:43.479927 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:43.482296 (+ 2369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:43.482306 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:43.482321 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:50:43.482345 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:43.484188 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:43.484747 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:43.484753 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:43.484755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:43.484848 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:43.485222 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:43.485226 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:43.485227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:43.485525 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:43.486437 (+ 912us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:43.486551 (+ 114us) server_negotiation.cc:299] Negotiation successful
0927 12:50:43.486706 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":133,"threads_started":1}
I20260927 12:50:43.488727 23106 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41253 } has no permanent_uuid. Determining permanent_uuid...
I20260927 12:50:43.489073 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:43.481028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37335 (local address 127.0.0.1:56018)
0927 12:50:43.481125 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:43.481138 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:50:43.481203 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:50:43.481733 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:50:43.481737 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:50:43.481905 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:50:43.482024 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:43.482033 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:43.482875 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:43.482878 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:43.483425 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:43.483434 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:43.483879 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:43.483883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:43.483988 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:43.484340 (+ 352us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:50:43.484353 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:50:43.485466 (+ 1113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:50:43.487891 (+ 2425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:50:43.487894 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:50:43.487896 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:50:43.488061 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:50:43.488276 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:50:43.488279 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:50:43.488282 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:50:43.488354 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:50:43.488650 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:50:43.488652 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:50:43.488726 (+ 74us) client_negotiation.cc:769] Sending connection context
0927 12:50:43.488817 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 12:50:43.488933 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":4}
I20260927 12:50:43.489470 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:43.489189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41253 (local address 127.0.0.1:33048)
0927 12:50:43.489367 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:43.489410 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260927 12:50:43.489902 23187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:43.481236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56018 (local address 127.0.98.61:37335)
0927 12:50:43.481548 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:43.481559 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:43.481569 (+ 10us) server_negotiation.cc:407] Connection header received
0927 12:50:43.481593 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:43.481595 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:43.481625 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:43.481699 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:43.482198 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:43.482780 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:43.483539 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:43.483791 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:43.483878 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:43.485577 (+ 1699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:43.485581 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:43.485584 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:50:43.485602 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:43.487801 (+ 2199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:43.488135 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:43.488138 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:43.488140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:43.488205 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:43.488421 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:43.488424 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:43.488425 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:43.488573 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:43.488652 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:43.489631 (+ 979us) server_negotiation.cc:299] Negotiation successful
0927 12:50:43.489738 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":259,"thread_start_us":131,"threads_started":1}
I20260927 12:50:43.490243 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:43.489959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41253 (local address 127.0.0.1:33060)
0927 12:50:43.490128 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:43.490179 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
W20260927 12:50:43.490411 23106 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41253: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41253: connect: Connection refused (error 111)
I20260927 12:50:43.490496 23181 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41253 } has no permanent_uuid. Determining permanent_uuid...
I20260927 12:50:43.491081 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:43.490868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41253 (local address 127.0.0.1:33076)
0927 12:50:43.490965 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:43.491031 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260927 12:50:43.491255 23130 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41253: connect: Connection refused (error 111)
I20260927 12:50:43.492025 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:43.491841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41253 (local address 127.0.0.1:33084)
0927 12:50:43.491937 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:43.491979 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260927 12:50:43.492134 23181 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41253: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41253: connect: Connection refused (error 111)
I20260927 12:50:43.521566 23106 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41253 } attempt: 1
I20260927 12:50:43.522487 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:43.522168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41253 (local address 127.0.0.1:33094)
0927 12:50:43.522338 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:43.522401 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 12:50:43.523180 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:43.522954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41253 (local address 127.0.0.1:33098)
0927 12:50:43.523051 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:43.523096 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260927 12:50:43.523365 23106 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41253: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41253: connect: Connection refused (error 111)
I20260927 12:50:43.541374 23181 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41253 } attempt: 1
I20260927 12:50:43.542191 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:43.541879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41253 (local address 127.0.0.1:33100)
0927 12:50:43.542014 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:43.542109 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 12:50:43.542985 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:43.542753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41253 (local address 127.0.0.1:33108)
0927 12:50:43.542881 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:43.542928 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260927 12:50:43.543119 23181 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41253: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41253: connect: Connection refused (error 111)
I20260927 12:50:43.598553 23106 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41253 } attempt: 2
I20260927 12:50:43.599367 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:43.599084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41253 (local address 127.0.0.1:33114)
0927 12:50:43.599223 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:43.599285 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 12:50:43.600059 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:43.599862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41253 (local address 127.0.0.1:33122)
0927 12:50:43.599955 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:43.600001 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260927 12:50:43.600178 23106 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41253: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41253: connect: Connection refused (error 111)
W20260927 12:50:43.600708 23188 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 12:50:43.601001 23188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:50:43.601042 23188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:50:43.601105 23188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:50:43.605278 23188 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:50:43.605427 23188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:50:43.605499 23188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:50:43.605532 23188 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:50:43.605563 23188 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:50:43.611270 23181 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41253 } attempt: 2
I20260927 12:50:43.611547 23188 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38661,127.0.98.61:37335,127.0.98.60:41253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:50:43.612106 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:43.611779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41253 (local address 127.0.0.1:33130)
0927 12:50:43.611943 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:43.612017 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 12:50:43.612838 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:43.612609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41253 (local address 127.0.0.1:33144)
0927 12:50:43.612720 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:43.612773 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260927 12:50:43.612924 23188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
W20260927 12:50:43.613013 23181 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41253: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41253: connect: Connection refused (error 111)
I20260927 12:50:43.613984 23188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:50:43.622496 23197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:50:43.622531 23194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:50:43.622496 23195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:50:43.622885 23188 server_base.cc:1034] running on GCE node
I20260927 12:50:43.623476 23188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:50:43.624652 23188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:50:43.625837 23188 hybrid_clock.cc:648] HybridClock initialized: now 1790513443625802 us; error 53 us; skew 500 ppm
Sep 27 12:50:43 dist-test-slave-lkn4 krb5kdc[23025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:50:43.628854 23188 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 12:50:43.629832 23188 webserver.cc:460] Webserver started at http://127.0.98.60:41593/ using document root <none> and password file <none>
I20260927 12:50:43.630425 23188 fs_manager.cc:359] Metadata directory not provided
I20260927 12:50:43.630497 23188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:50:43.630713 23188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:50:43.632839 23188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/minicluster-data/master-2/data/instance:
uuid: "200646d3c99f4626a64f025e81a65073"
format_stamp: "Formatted at 2026-09-27 12:50:43 on dist-test-slave-lkn4"
server_key: "6d41e723e8572a92f1a6f4836676592c"
server_key_iv: "bc97793052a8c1810e887c5a4adcd164"
server_key_version: "encryptionkey@0"
I20260927 12:50:43.633412 23188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/minicluster-data/master-2/wal/instance:
uuid: "200646d3c99f4626a64f025e81a65073"
format_stamp: "Formatted at 2026-09-27 12:50:43 on dist-test-slave-lkn4"
server_key: "6d41e723e8572a92f1a6f4836676592c"
server_key_iv: "bc97793052a8c1810e887c5a4adcd164"
server_key_version: "encryptionkey@0"
I20260927 12:50:43.637328 23188 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 12:50:43.640082 23204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:50:43.641403 23188 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:50:43.641521 23188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/minicluster-data/master-2/wal
uuid: "200646d3c99f4626a64f025e81a65073"
format_stamp: "Formatted at 2026-09-27 12:50:43 on dist-test-slave-lkn4"
server_key: "6d41e723e8572a92f1a6f4836676592c"
server_key_iv: "bc97793052a8c1810e887c5a4adcd164"
server_key_version: "encryptionkey@0"
I20260927 12:50:43.641620 23188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:50:43.652510 23188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:50:43.655514 23188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:50:43.655699 23188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:50:43.662550 23188 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:41253
I20260927 12:50:43.662868 23256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:41253 every 8 connection(s)
I20260927 12:50:43.663555 23188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/minicluster-data/master-2/data/info.pb
I20260927 12:50:43.666448 23257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:50:43.670418 23257 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38661 } has no permanent_uuid. Determining permanent_uuid...
I20260927 12:50:43.672741 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu as pid 23188
I20260927 12:50:43.672865 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTC8Et9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790513161160093-392-0/minicluster-data/master-2/wal/instance
I20260927 12:50:43.673168 392 external_mini_cluster.cc:1428] Setting key 476bcd09c27d00b8db8cdea94c5c7306
Sep 27 12:50:43 dist-test-slave-lkn4 krb5kdc[23025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 27 12:50:43 dist-test-slave-lkn4 krb5kdc[23025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 12:50:43.684729 23110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:43.672386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46776 (local address 127.0.98.62:38661)
0927 12:50:43.672539 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:43.672545 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:43.673444 (+ 899us) server_negotiation.cc:407] Connection header received
0927 12:50:43.674609 (+ 1165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:43.674612 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:43.674661 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:43.674746 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:43.675838 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:43.676601 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:43.678001 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:43.678369 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:43.678472 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:43.681201 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:43.681205 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:43.681208 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:50:43.681239 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:43.682716 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:43.683272 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:43.683275 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:43.683276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:43.683354 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:43.683656 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:43.683659 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:43.683661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:43.683854 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:43.683955 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:43.684439 (+ 484us) server_negotiation.cc:299] Negotiation successful
0927 12:50:43.684561 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 12:50:43.685663 23259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:43.672775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38661 (local address 127.0.0.1:46776)
0927 12:50:43.673279 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:43.673334 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 12:50:43.674391 (+ 1057us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:50:43.674873 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:50:43.674879 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:50:43.675310 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:50:43.675657 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:43.675677 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:43.676704 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:43.676706 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:43.677877 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:43.677888 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:43.678516 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:43.678519 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:43.678657 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:43.679363 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:50:43.679385 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:50:43.681006 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:50:43.682815 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:50:43.682830 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:50:43.682841 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:50:43.683157 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:50:43.683435 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:50:43.683438 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:50:43.683440 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:50:43.683573 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:50:43.683950 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:50:43.683957 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:50:43.684215 (+ 258us) client_negotiation.cc:769] Sending connection context
0927 12:50:43.684394 (+ 179us) client_negotiation.cc:240] Negotiation successful
0927 12:50:43.684708 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":23,"spinlock_wait_cycles":896,"thread_start_us":122,"threads_started":1}
I20260927 12:50:43.685664 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:43.673749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33152 (local address 127.0.98.60:41253)
0927 12:50:43.674105 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:43.674117 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:43.674168 (+ 51us) server_negotiation.cc:407] Connection header received
0927 12:50:43.674416 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:43.674442 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:43.674687 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:43.674867 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:43.675681 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:43.676880 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:43.677997 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:43.678584 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:43.678687 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:43.681556 (+ 2869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:43.681562 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:43.681574 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:50:43.681594 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:43.683141 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:43.683522 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:43.683527 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:43.683528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:43.683599 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:43.683839 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:43.683842 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:43.683844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:43.684213 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:43.684402 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:43.684700 (+ 298us) server_negotiation.cc:299] Negotiation successful
0927 12:50:43.684847 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":238,"thread_start_us":137,"threads_started":1}
I20260927 12:50:43.686964 23257 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37335 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 12:50:43 dist-test-slave-lkn4 krb5kdc[23025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 27 12:50:43 dist-test-slave-lkn4 krb5kdc[23025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:50:43.695679 23187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:43.687639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56026 (local address 127.0.98.61:37335)
0927 12:50:43.687787 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:43.687792 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:43.687808 (+ 16us) server_negotiation.cc:407] Connection header received
0927 12:50:43.687896 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:43.687898 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:43.687942 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:43.688019 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:43.688478 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:43.689056 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:43.689833 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:43.690150 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:43.690257 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:43.692431 (+ 2174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:43.692434 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:43.692436 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:50:43.692464 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:43.693867 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:43.694421 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:43.694424 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:43.694425 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:43.694504 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:43.694800 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:43.694804 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:43.694806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:43.695062 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:43.695183 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:43.695442 (+ 259us) server_negotiation.cc:299] Negotiation successful
0927 12:50:43.695561 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 12:50:43.695674 23259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:43.687533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37335 (local address 127.0.0.1:56026)
0927 12:50:43.687718 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:43.687732 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:50:43.687804 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:50:43.688052 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:50:43.688056 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:50:43.688234 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:50:43.688348 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:43.688354 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:43.689175 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:43.689179 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:43.689718 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:43.689727 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:43.690256 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:43.690260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:43.690367 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:43.690865 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:50:43.690882 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:50:43.692284 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:50:43.693993 (+ 1709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:50:43.693995 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:50:43.693997 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:50:43.694320 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:50:43.694587 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:50:43.694590 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:50:43.694593 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:50:43.694680 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:50:43.695143 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:50:43.695146 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:50:43.695244 (+ 98us) client_negotiation.cc:769] Sending connection context
0927 12:50:43.695370 (+ 126us) client_negotiation.cc:240] Negotiation successful
0927 12:50:43.695494 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112}
I20260927 12:50:43.696475 23110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:43.688079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46780 (local address 127.0.98.62:38661)
0927 12:50:43.688177 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:43.688181 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:43.688278 (+ 97us) server_negotiation.cc:407] Connection header received
0927 12:50:43.688379 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:43.688382 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:43.688422 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:43.688481 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:43.689041 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:43.689514 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:43.690291 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:43.690501 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:43.690583 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:43.692963 (+ 2380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:43.692966 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:43.692968 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:50:43.692991 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:43.694688 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:43.695143 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:43.695146 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:43.695148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:43.695227 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:43.695483 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:43.695486 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:43.695488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:43.695683 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:43.695892 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:43.696169 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 12:50:43.696340 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 12:50:43.696666 23257 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41253 } has no permanent_uuid. Determining permanent_uuid...
I20260927 12:50:43.699316 23106 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41253 } attempt: 3
Sep 27 12:50:43 dist-test-slave-lkn4 krb5kdc[23025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 12:50:43 dist-test-slave-lkn4 krb5kdc[23025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 12:50:43.704181 23181 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41253 } attempt: 3
Sep 27 12:50:43 dist-test-slave-lkn4 krb5kdc[23025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 12:50:43.706789 23259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:43.697184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41253 (local address 127.0.0.1:33160)
0927 12:50:43.697287 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:43.697300 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:50:43.697375 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:50:43.697639 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:50:43.697641 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:50:43.697803 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:50:43.697890 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:43.697897 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:43.698750 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:43.698753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:43.699452 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:43.699462 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:43.700296 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:43.700300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:43.700376 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:43.700816 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:50:43.700829 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:50:43.702310 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:50:43.705107 (+ 2797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:50:43.705110 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:50:43.705112 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:50:43.705346 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:50:43.705775 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:50:43.705778 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:50:43.705780 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:50:43.705872 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:50:43.706299 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:50:43.706301 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:50:43.706374 (+ 73us) client_negotiation.cc:769] Sending connection context
0927 12:50:43.706504 (+ 130us) client_negotiation.cc:240] Negotiation successful
0927 12:50:43.706621 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 12:50:43.706880 23187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:43.697513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56042 (local address 127.0.98.61:37335)
0927 12:50:43.697606 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:43.697610 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:43.697647 (+ 37us) server_negotiation.cc:407] Connection header received
0927 12:50:43.697760 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:43.697762 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:43.697794 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:43.697851 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:43.698469 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:43.699047 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:43.699986 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:43.700322 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:43.700425 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:43.702939 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:43.702942 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:43.702945 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:50:43.702970 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:43.704996 (+ 2026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:43.705461 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:43.705465 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:43.705467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:43.705549 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:43.705822 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:43.705825 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:43.705827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:43.706080 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:43.706256 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:43.706606 (+ 350us) server_negotiation.cc:299] Negotiation successful
0927 12:50:43.706730 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 12:50:43.707301 23261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:43.697206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33160 (local address 127.0.98.60:41253)
0927 12:50:43.697301 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:43.697305 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:43.697340 (+ 35us) server_negotiation.cc:407] Connection header received
0927 12:50:43.697451 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:43.697454 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:43.697489 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:43.697550 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:43.698000 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:43.698647 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:43.699838 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:43.700188 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:43.700285 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:43.702486 (+ 2201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:43.702491 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:43.702494 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:50:43.702517 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:43.705001 (+ 2484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:43.705426 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:43.705430 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:43.705432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:43.705503 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:43.705970 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:43.705974 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:43.705975 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:43.706193 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:43.706952 (+ 759us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:43.707018 (+ 66us) server_negotiation.cc:299] Negotiation successful
0927 12:50:43.707112 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260927 12:50:43.708514 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:43.699702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41253 (local address 127.0.0.1:33166)
0927 12:50:43.699822 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:43.699835 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:50:43.699909 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:50:43.700350 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:50:43.700353 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:50:43.700522 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:50:43.700613 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:43.700618 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:43.701581 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:43.701584 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:43.702168 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:43.702179 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:43.702763 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:43.702779 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:43.702862 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:43.703272 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:50:43.703286 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:50:43.704659 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:50:43.706480 (+ 1821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:50:43.706483 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:50:43.706485 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:50:43.706707 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:50:43.707572 (+ 865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:50:43.707576 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:50:43.707578 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:50:43.707669 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:50:43.708072 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:50:43.708075 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:50:43.708166 (+ 91us) client_negotiation.cc:769] Sending connection context
0927 12:50:43.708241 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 12:50:43.708373 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260927 12:50:43.709362 23262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:43.699727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33166 (local address 127.0.98.60:41253)
0927 12:50:43.700142 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:43.700149 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:43.700158 (+ 9us) server_negotiation.cc:407] Connection header received
0927 12:50:43.700182 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:43.700184 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:43.700215 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:43.700695 (+ 480us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:43.700799 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:43.701487 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:43.702296 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:43.702633 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:43.703051 (+ 418us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:43.704802 (+ 1751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:43.704805 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:43.704807 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:50:43.704826 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:43.706368 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:43.707199 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:43.707202 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:43.707204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:43.707275 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:43.707783 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:43.707786 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:43.707788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:43.707965 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:43.708962 (+ 997us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:43.709082 (+ 120us) server_negotiation.cc:299] Negotiation successful
0927 12:50:43.709192 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":117,"threads_started":1}
I20260927 12:50:43.710991 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:50:43.711159 23257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 200646d3c99f4626a64f025e81a65073: Bootstrap starting.
I20260927 12:50:43.711925 23106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf0c1b8cfb65477fadd3db86d7af257d: Bootstrap starting.
I20260927 12:50:43.712141 23080 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "bf0c1b8cfb65477fadd3db86d7af257d"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46780
I20260927 12:50:43.713963 23106 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf0c1b8cfb65477fadd3db86d7af257d: Neither blocks nor log segments found. Creating new log.
I20260927 12:50:43.713989 23257 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 200646d3c99f4626a64f025e81a65073: Neither blocks nor log segments found. Creating new log.
I20260927 12:50:43.714780 23106 log.cc:824] T 00000000000000000000000000000000 P bf0c1b8cfb65477fadd3db86d7af257d: Log is configured to *not* fsync() on all Append() calls
I20260927 12:50:43.714859 23257 log.cc:824] T 00000000000000000000000000000000 P 200646d3c99f4626a64f025e81a65073: Log is configured to *not* fsync() on all Append() calls
Sep 27 12:50:43 dist-test-slave-lkn4 krb5kdc[23025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 12:50:43.716871 23257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 200646d3c99f4626a64f025e81a65073: No bootstrap required, opened a new log
I20260927 12:50:43.716871 23106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf0c1b8cfb65477fadd3db86d7af257d: No bootstrap required, opened a new log
I20260927 12:50:43.719336 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:43.704562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41253 (local address 127.0.0.1:33178)
0927 12:50:43.704838 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0927 12:50:43.704852 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:50:43.704926 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:50:43.705965 (+ 1039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:50:43.705969 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:50:43.706170 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:50:43.706256 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:43.706262 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:43.707560 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:43.707563 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:43.708257 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:50:43.708268 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:50:43.713683 (+ 5415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:50:43.713707 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:50:43.713809 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:43.714364 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:50:43.714379 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:50:43.715802 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:50:43.717755 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:50:43.717759 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:50:43.717761 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:50:43.717998 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:50:43.718302 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:50:43.718305 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:50:43.718307 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:50:43.718393 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:50:43.718793 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:50:43.718796 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:50:43.718898 (+ 102us) client_negotiation.cc:769] Sending connection context
0927 12:50:43.718992 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 12:50:43.719169 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198}
I20260927 12:50:43.719362 23263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:43.704615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33178 (local address 127.0.98.60:41253)
0927 12:50:43.705758 (+ 1143us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:43.705766 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:43.705779 (+ 13us) server_negotiation.cc:407] Connection header received
0927 12:50:43.705811 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:43.705813 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:43.705852 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:43.706346 (+ 494us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:43.706445 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:43.707290 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:43.708425 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:43.709687 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:43.714090 (+ 4403us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:43.715945 (+ 1855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:43.715949 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:43.715952 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:50:43.715975 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:43.717646 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:43.718128 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:43.718132 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:43.718134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:43.718215 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:43.718485 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:43.718488 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:43.718490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:43.718698 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:43.718792 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:43.719069 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 12:50:43.719186 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1080,"thread_start_us":92,"threads_started":1}
I20260927 12:50:43.720042 23106 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf0c1b8cfb65477fadd3db86d7af257d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf0c1b8cfb65477fadd3db86d7af257d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38661 } } peers { permanent_uuid: "217eb45bb3e34dd58a4275a4e4bc24d0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37335 } } peers { permanent_uuid: "200646d3c99f4626a64f025e81a65073" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41253 } }
I20260927 12:50:43.720273 23106 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf0c1b8cfb65477fadd3db86d7af257d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf0c1b8cfb65477fadd3db86d7af257d, State: Initialized, Role: FOLLOWER
I20260927 12:50:43.720602 23106 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf0c1b8cfb65477fadd3db86d7af257d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf0c1b8cfb65477fadd3db86d7af257d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38661 } } peers { permanent_uuid: "217eb45bb3e34dd58a4275a4e4bc24d0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37335 } } peers { permanent_uuid: "200646d3c99f4626a64f025e81a65073" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41253 } }
I20260927 12:50:43.721095 23266 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf0c1b8cfb65477fadd3db86d7af257d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf0c1b8cfb65477fadd3db86d7af257d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38661 } } peers { permanent_uuid: "217eb45bb3e34dd58a4275a4e4bc24d0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37335 } } peers { permanent_uuid: "200646d3c99f4626a64f025e81a65073" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41253 } } }
I20260927 12:50:43.721272 23266 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf0c1b8cfb65477fadd3db86d7af257d [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:50:43.721408 23257 raft_consensus.cc:348] T 00000000000000000000000000000000 P 200646d3c99f4626a64f025e81a65073 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf0c1b8cfb65477fadd3db86d7af257d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38661 } } peers { permanent_uuid: "217eb45bb3e34dd58a4275a4e4bc24d0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37335 } } peers { permanent_uuid: "200646d3c99f4626a64f025e81a65073" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41253 } }
I20260927 12:50:43.721618 23257 raft_consensus.cc:729] T 00000000000000000000000000000000 P 200646d3c99f4626a64f025e81a65073 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 200646d3c99f4626a64f025e81a65073, State: Initialized, Role: FOLLOWER
I20260927 12:50:43.721931 23106 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf0c1b8cfb65477fadd3db86d7af257d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:50:43.721906 23257 consensus_queue.cc:260] T 00000000000000000000000000000000 P 200646d3c99f4626a64f025e81a65073 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf0c1b8cfb65477fadd3db86d7af257d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38661 } } peers { permanent_uuid: "217eb45bb3e34dd58a4275a4e4bc24d0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37335 } } peers { permanent_uuid: "200646d3c99f4626a64f025e81a65073" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41253 } }
I20260927 12:50:43.722002 23080 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf0c1b8cfb65477fadd3db86d7af257d [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 12:50:43.722112 23080 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf0c1b8cfb65477fadd3db86d7af257d [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:50:43.722539 23181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 217eb45bb3e34dd58a4275a4e4bc24d0: Bootstrap starting.
I20260927 12:50:43.723194 23257 sys_catalog.cc:564] T 00000000000000000000000000000000 P 200646d3c99f4626a64f025e81a65073 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:50:43.723322 23080 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf0c1b8cfb65477fadd3db86d7af257d [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf0c1b8cfb65477fadd3db86d7af257d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38661 } } peers { permanent_uuid: "217eb45bb3e34dd58a4275a4e4bc24d0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37335 } } peers { permanent_uuid: "200646d3c99f4626a64f025e81a65073" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41253 } }
I20260927 12:50:43.724321 23080 leader_election.cc:288] T 00000000000000000000000000000000 P bf0c1b8cfb65477fadd3db86d7af257d [CANDIDATE]: Term 1 election: Requested vote from peers 217eb45bb3e34dd58a4275a4e4bc24d0 (127.0.98.61:37335), 200646d3c99f4626a64f025e81a65073 (127.0.98.60:41253)
I20260927 12:50:43.725683 23181 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 217eb45bb3e34dd58a4275a4e4bc24d0: Neither blocks nor log segments found. Creating new log.
I20260927 12:50:43.726614 23181 log.cc:824] T 00000000000000000000000000000000 P 217eb45bb3e34dd58a4275a4e4bc24d0: Log is configured to *not* fsync() on all Append() calls
W20260927 12:50:43.727649 23279 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bf0c1b8cfb65477fadd3db86d7af257d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 12:50:43.727720 23279 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260927 12:50:43.728225 23289 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 200646d3c99f4626a64f025e81a65073: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260927 12:50:43.728193 23231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bf0c1b8cfb65477fadd3db86d7af257d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "200646d3c99f4626a64f025e81a65073"
W20260927 12:50:43.728295 23289 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260927 12:50:43.728375 23059 tablet.cc:2369] T 00000000000000000000000000000000 P bf0c1b8cfb65477fadd3db86d7af257d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:50:43.728399 23231 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 200646d3c99f4626a64f025e81a65073 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:50:43.729697 23231 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 200646d3c99f4626a64f025e81a65073 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf0c1b8cfb65477fadd3db86d7af257d in term 1.
I20260927 12:50:43.730808 23055 leader_election.cc:302] T 00000000000000000000000000000000 P bf0c1b8cfb65477fadd3db86d7af257d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 200646d3c99f4626a64f025e81a65073, bf0c1b8cfb65477fadd3db86d7af257d; no voters:
I20260927 12:50:43.731925 23181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 217eb45bb3e34dd58a4275a4e4bc24d0: No bootstrap required, opened a new log
I20260927 12:50:43.733600 23266 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf0c1b8cfb65477fadd3db86d7af257d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:50:43.730688 23038 master_runner.cc:186] Error getting master registration for 127.0.98.61:37335: OK, instance_id { permanent_uuid: "217eb45bb3e34dd58a4275a4e4bc24d0" instance_seqno: 1790513443452459 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 12:50:43.733835 23266 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf0c1b8cfb65477fadd3db86d7af257d [term 1 LEADER]: Becoming Leader. State: Replica: bf0c1b8cfb65477fadd3db86d7af257d, State: Running, Role: LEADER
I20260927 12:50:43.733911 23038 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 12:50:43.734236 23266 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf0c1b8cfb65477fadd3db86d7af257d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf0c1b8cfb65477fadd3db86d7af257d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38661 } } peers { permanent_uuid: "217eb45bb3e34dd58a4275a4e4bc24d0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37335 } } peers { permanent_uuid: "200646d3c99f4626a64f025e81a65073" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41253 } }
I20260927 12:50:43.735934 23181 raft_consensus.cc:348] T 00000000000000000000000000000000 P 217eb45bb3e34dd58a4275a4e4bc24d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf0c1b8cfb65477fadd3db86d7af257d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38661 } } peers { permanent_uuid: "217eb45bb3e34dd58a4275a4e4bc24d0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37335 } } peers { permanent_uuid: "200646d3c99f4626a64f025e81a65073" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41253 } }
I20260927 12:50:43.736043 23266 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf0c1b8cfb65477fadd3db86d7af257d [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf0c1b8cfb65477fadd3db86d7af257d. Latest consensus state: current_term: 1 leader_uuid: "bf0c1b8cfb65477fadd3db86d7af257d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf0c1b8cfb65477fadd3db86d7af257d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38661 } } peers { permanent_uuid: "217eb45bb3e34dd58a4275a4e4bc24d0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37335 } } peers { permanent_uuid: "200646d3c99f4626a64f025e81a65073" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41253 } } }
I20260927 12:50:43.736176 23266 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf0c1b8cfb65477fadd3db86d7af257d [sys.catalog]: This master's current role is: LEADER
I20260927 12:50:43.736167 23181 raft_consensus.cc:729] T 00000000000000000000000000000000 P 217eb45bb3e34dd58a4275a4e4bc24d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 217eb45bb3e34dd58a4275a4e4bc24d0, State: Initialized, Role: FOLLOWER
I20260927 12:50:43.736490 23181 consensus_queue.cc:260] T 00000000000000000000000000000000 P 217eb45bb3e34dd58a4275a4e4bc24d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf0c1b8cfb65477fadd3db86d7af257d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38661 } } peers { permanent_uuid: "217eb45bb3e34dd58a4275a4e4bc24d0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37335 } } peers { permanent_uuid: "200646d3c99f4626a64f025e81a65073" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41253 } }
I20260927 12:50:43.734129 23188 master_runner.cc:186] Error getting master registration for 127.0.98.61:37335: OK, instance_id { permanent_uuid: "217eb45bb3e34dd58a4275a4e4bc24d0" instance_seqno: 1790513443452459 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 12:50:43.737288 23188 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 12:50:43.737977 23181 sys_catalog.cc:564] T 00000000000000000000000000000000 P 217eb45bb3e34dd58a4275a4e4bc24d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:50:43.737993 23155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bf0c1b8cfb65477fadd3db86d7af257d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "217eb45bb3e34dd58a4275a4e4bc24d0"
I20260927 12:50:43.738261 23155 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 217eb45bb3e34dd58a4275a4e4bc24d0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:50:43.739677 23155 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 217eb45bb3e34dd58a4275a4e4bc24d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf0c1b8cfb65477fadd3db86d7af257d in term 1.
I20260927 12:50:43.740890 23300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 217eb45bb3e34dd58a4275a4e4bc24d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf0c1b8cfb65477fadd3db86d7af257d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38661 } } peers { permanent_uuid: "217eb45bb3e34dd58a4275a4e4bc24d0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37335 } } peers { permanent_uuid: "200646d3c99f4626a64f025e81a65073" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41253 } } }
I20260927 12:50:43.741093 23300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 217eb45bb3e34dd58a4275a4e4bc24d0 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:50:43.741109 23299 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:50:43.742174 23299 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:50:43.745569 23267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 200646d3c99f4626a64f025e81a65073 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf0c1b8cfb65477fadd3db86d7af257d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38661 } } peers { permanent_uuid: "217eb45bb3e34dd58a4275a4e4bc24d0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37335 } } peers { permanent_uuid: "200646d3c99f4626a64f025e81a65073" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41253 } } }
I20260927 12:50:43.745702 23267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 200646d3c99f4626a64f025e81a65073 [sys.catalog]: This master's current role is: FOLLOWER
W20260927 12:50:43.746630 23313 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 217eb45bb3e34dd58a4275a4e4bc24d0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 12:50:43.746752 23313 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 12:50:43.753616 23187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:43.736745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56044 (local address 127.0.98.61:37335)
0927 12:50:43.737444 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:43.737449 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:43.744529 (+ 7080us) server_negotiation.cc:407] Connection header received
0927 12:50:43.744694 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:43.744697 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:43.744748 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:43.744837 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:43.745822 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:43.746538 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:43.747634 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:43.747990 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:43.748058 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:43.749894 (+ 1836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:43.749899 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:43.749901 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:50:43.749929 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:43.751668 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:43.752182 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:43.752186 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:43.752188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:43.752273 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:43.752599 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:43.752603 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:43.752605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:43.752845 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:43.752962 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:43.753319 (+ 357us) server_negotiation.cc:299] Negotiation successful
0927 12:50:43.753443 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614}
I20260927 12:50:43.763131 23110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:43.737448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46796 (local address 127.0.98.62:38661)
0927 12:50:43.737546 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:43.737550 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:43.744749 (+ 7199us) server_negotiation.cc:407] Connection header received
0927 12:50:43.744912 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:43.744916 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:43.744964 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:43.745014 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:43.747064 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:43.747836 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:43.750142 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:43.750490 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:43.750572 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:43.758292 (+ 7720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:43.758297 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:43.758309 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:50:43.758342 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:43.759969 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:43.760451 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:43.760455 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:43.760457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:43.760543 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:43.762144 (+ 1601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:43.762147 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:43.762149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:43.762376 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:43.762490 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:43.762831 (+ 341us) server_negotiation.cc:299] Negotiation successful
0927 12:50:43.762961 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 12:50:43.763322 23263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:43.738153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33186 (local address 127.0.98.60:41253)
0927 12:50:43.738266 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:43.738271 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:43.754176 (+ 15905us) server_negotiation.cc:407] Connection header received
0927 12:50:43.754365 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:43.754369 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:43.754422 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:43.754472 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:43.755224 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:43.756033 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:43.757156 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:43.757531 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:43.757610 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:43.759897 (+ 2287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:50:43.759901 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:50:43.759904 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:50:43.759935 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:50:43.761507 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:43.761991 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:43.761995 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:43.761997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:43.762111 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:50:43.762394 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:50:43.762397 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:50:43.762400 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:50:43.762604 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:50:43.762698 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:43.763040 (+ 342us) server_negotiation.cc:299] Negotiation successful
0927 12:50:43.763162 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 12:50:43.765846 23231 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 200646d3c99f4626a64f025e81a65073 [term 1 FOLLOWER]: Refusing update from remote peer bf0c1b8cfb65477fadd3db86d7af257d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:50:43.765882 23155 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 217eb45bb3e34dd58a4275a4e4bc24d0 [term 1 FOLLOWER]: Refusing update from remote peer bf0c1b8cfb65477fadd3db86d7af257d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:50:43.766547 23297 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bf0c1b8cfb65477fadd3db86d7af257d [LEADER]: Connected to new peer: Peer: permanent_uuid: "200646d3c99f4626a64f025e81a65073" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:50:43.766726 23266 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bf0c1b8cfb65477fadd3db86d7af257d [LEADER]: Connected to new peer: Peer: permanent_uuid: "217eb45bb3e34dd58a4275a4e4bc24d0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:50:43.770987 23267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 200646d3c99f4626a64f025e81a65073 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf0c1b8cfb65477fadd3db86d7af257d. Latest consensus state: current_term: 1 leader_uuid: "bf0c1b8cfb65477fadd3db86d7af257d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf0c1b8cfb65477fadd3db86d7af257d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38661 } } peers { permanent_uuid: "217eb45bb3e34dd58a4275a4e4bc24d0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37335 } } peers { permanent_uuid: "200646d3c99f4626a64f025e81a65073" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41253 } } }
I20260927 12:50:43.771118 23267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 200646d3c99f4626a64f025e81a65073 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:50:43.771064 23300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 217eb45bb3e34dd58a4275a4e4bc24d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf0c1b8cfb65477fadd3db86d7af257d. Latest consensus state: current_term: 1 leader_uuid: "bf0c1b8cfb65477fadd3db86d7af257d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf0c1b8cfb65477fadd3db86d7af257d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38661 } } peers { permanent_uuid: "217eb45bb3e34dd58a4275a4e4bc24d0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37335 } } peers { permanent_uuid: "200646d3c99f4626a64f025e81a65073" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41253 } } }
I20260927 12:50:43.771186 23300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 217eb45bb3e34dd58a4275a4e4bc24d0 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:50:43.774417 23297 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf0c1b8cfb65477fadd3db86d7af257d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bf0c1b8cfb65477fadd3db86d7af257d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf0c1b8cfb65477fadd3db86d7af257d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38661 } } peers { permanent_uuid: "217eb45bb3e34dd58a4275a4e4bc24d0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37335 } } peers { permanent_uuid: "200646d3c99f4626a64f025e81a65073" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41253 } } }
I20260927 12:50:43.774557 23297 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf0c1b8cfb65477fadd3db86d7af257d [sys.catalog]: This master's current role is: LEADER
I20260927 12:50:43.775965 23299 catalog_manager.cc:1347] Generated new cluster ID: 64a27e5bb0d2494d88584da4b6aa3fa0
I20260927 12:50:43.775928 23267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 200646d3c99f4626a64f025e81a65073 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bf0c1b8cfb65477fadd3db86d7af257d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf0c1b8cfb65477fadd3db86d7af257d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38661 } } peers { permanent_uuid: "217eb45bb3e34dd58a4275a4e4bc24d0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37335 } } peers { permanent_uuid: "200646d3c99f4626a64f025e81a65073" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41253 } } }
I20260927 12:50:43.776028 23299 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:50:43.776047 23267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 200646d3c99f4626a64f025e81a65073 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:50:43.784451 23266 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf0c1b8cfb65477fadd3db86d7af257d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bf0c1b8cfb65477fadd3db86d7af257d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf0c1b8cfb65477fadd3db86d7af257d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38661 } } peers { permanent_uuid: "217eb45bb3e34dd58a4275a4e4bc24d0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37335 } } peers { permanent_uuid: "200646d3c99f4626a64f025e81a65073" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41253 } } }
I20260927 12:50:43.784595 23266 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf0c1b8cfb65477fadd3db86d7af257d [sys.catalog]: This master's current role is: LEADER
I20260927 12:50:43.785530 23317 mvcc.cc:205] Tried to move back new op lower bound from 7333943065657819136 to 7333943065538904064. Current Snapshot: MvccSnapshot[applied={T|T < 7333943065657819136}]
I20260927 12:50:43.785712 23300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 217eb45bb3e34dd58a4275a4e4bc24d0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bf0c1b8cfb65477fadd3db86d7af257d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf0c1b8cfb65477fadd3db86d7af257d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38661 } } peers { permanent_uuid: "217eb45bb3e34dd58a4275a4e4bc24d0" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37335 } } peers { permanent_uuid: "200646d3c99f4626a64f025e81a65073" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41253 } } }
I20260927 12:50:43.785811 23300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 217eb45bb3e34dd58a4275a4e4bc24d0 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:50:43.788053 23318 mvcc.cc:205] Tried to move back new op lower bound from 7333943065657819136 to 7333943065538904064. Current Snapshot: MvccSnapshot[applied={T|T < 7333943065657819136}]
I20260927 12:50:43.803356 23299 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:50:43.804714 23299 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:50:43.821126 23299 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf0c1b8cfb65477fadd3db86d7af257d: Generated new TSK 0
I20260927 12:50:43.822736 23299 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 12:50:43.859283 23187 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:50:43.855966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56052 (local address 127.0.98.61:37335)
0927 12:50:43.856104 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:43.856109 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:43.858139 (+ 2030us) server_negotiation.cc:407] Connection header received
0927 12:50:43.858351 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:43.858355 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:43.858409 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:43.858460 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:43.859135 (+ 675us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56052: BlockingRecv error: recv got EOF from 127.0.0.1:56052 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260927 12:50:43.860739 23110 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:50:43.856871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46800 (local address 127.0.98.62:38661)
0927 12:50:43.856989 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:43.856993 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:43.859791 (+ 2798us) server_negotiation.cc:407] Connection header received
0927 12:50:43.859983 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:43.859986 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:43.860035 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:43.860083 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:43.860596 (+ 513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46800: BlockingRecv error: recv got EOF from 127.0.0.1:46800 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260927 12:50:43.862190 23263 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:50:43.857528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33192 (local address 127.0.98.60:41253)
0927 12:50:43.857650 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:43.857654 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:43.861167 (+ 3513us) server_negotiation.cc:407] Connection header received
0927 12:50:43.861437 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:43.861440 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:43.861492 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:43.861539 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:43.862067 (+ 528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33192: BlockingRecv error: recv got EOF from 127.0.0.1:33192 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260927 12:50:43.868209 23187 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:50:43.866388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56064 (local address 127.0.98.61:37335)
0927 12:50:43.866537 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:43.866542 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:43.867201 (+ 659us) server_negotiation.cc:407] Connection header received
0927 12:50:43.867425 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:43.867429 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:43.867507 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:43.867556 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:43.868103 (+ 547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56064: BlockingRecv error: recv got EOF from 127.0.0.1:56064 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260927 12:50:43.873077 23187 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:50:43.871551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56068 (local address 127.0.98.61:37335)
0927 12:50:43.871663 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:43.871667 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:43.872165 (+ 498us) server_negotiation.cc:407] Connection header received
0927 12:50:43.872347 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:43.872350 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:43.872419 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:43.872468 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:43.872968 (+ 500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56068: BlockingRecv error: recv got EOF from 127.0.0.1:56068 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260927 12:50:43.878531 23187 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:50:43.877030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56080 (local address 127.0.98.61:37335)
0927 12:50:43.877134 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:43.877138 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:43.877625 (+ 487us) server_negotiation.cc:407] Connection header received
0927 12:50:43.877821 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:43.877825 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:43.877872 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:43.877921 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:43.878427 (+ 506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56080: BlockingRecv error: recv got EOF from 127.0.0.1:56080 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260927 12:50:43.886466 23187 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:50:43.884657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56086 (local address 127.0.98.61:37335)
0927 12:50:43.884812 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:43.884817 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:43.885379 (+ 562us) server_negotiation.cc:407] Connection header received
0927 12:50:43.885600 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:43.885605 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:43.885680 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:43.885749 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:43.886345 (+ 596us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56086: BlockingRecv error: recv got EOF from 127.0.0.1:56086 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260927 12:50:43.898836 23187 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:50:43.896998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56088 (local address 127.0.98.61:37335)
0927 12:50:43.897125 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:43.897130 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:43.897782 (+ 652us) server_negotiation.cc:407] Connection header received
0927 12:50:43.897998 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:43.898001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:43.898083 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:43.898139 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:43.898716 (+ 577us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56088: BlockingRecv error: recv got EOF from 127.0.0.1:56088 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260927 12:50:43.919351 23187 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:50:43.917367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56092 (local address 127.0.98.61:37335)
0927 12:50:43.917502 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:43.917507 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:43.918222 (+ 715us) server_negotiation.cc:407] Connection header received
0927 12:50:43.918454 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:43.918458 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:43.918518 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:43.918572 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:43.919221 (+ 649us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56092: BlockingRecv error: recv got EOF from 127.0.0.1:56092 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260927 12:50:43.955349 23187 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:50:43.953933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56102 (local address 127.0.98.61:37335)
0927 12:50:43.954098 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:43.954103 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:43.954389 (+ 286us) server_negotiation.cc:407] Connection header received
0927 12:50:43.954550 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:43.954554 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:43.954609 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:43.954688 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:43.955218 (+ 530us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56102: BlockingRecv error: recv got EOF from 127.0.0.1:56102 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260927 12:50:44.022413 23187 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:50:44.021286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56118 (local address 127.0.98.61:37335)
0927 12:50:44.021456 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:44.021461 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:44.021549 (+ 88us) server_negotiation.cc:407] Connection header received
0927 12:50:44.021687 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:44.021691 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:44.021745 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:44.021864 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:44.022293 (+ 429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56118: BlockingRecv error: recv got EOF from 127.0.0.1:56118 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260927 12:50:44.153698 23187 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:50:44.152540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56120 (local address 127.0.98.61:37335)
0927 12:50:44.152699 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:44.152704 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:44.152854 (+ 150us) server_negotiation.cc:407] Connection header received
0927 12:50:44.153004 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:44.153007 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:44.153057 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:44.153123 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:44.153580 (+ 457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56120: BlockingRecv error: recv got EOF from 127.0.0.1:56120 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260927 12:50:44.413286 23187 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:50:44.411944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56122 (local address 127.0.98.61:37335)
0927 12:50:44.412146 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:44.412151 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:44.412291 (+ 140us) server_negotiation.cc:407] Connection header received
0927 12:50:44.412458 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:44.412462 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:44.412513 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:44.412584 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:50:44.413123 (+ 539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56122: BlockingRecv error: recv got EOF from 127.0.0.1:56122 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
I20260927 12:50:44.729499 23289 catalog_manager.cc:1259] Loaded cluster ID: 64a27e5bb0d2494d88584da4b6aa3fa0
I20260927 12:50:44.729581 23289 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 200646d3c99f4626a64f025e81a65073: loading cluster ID for follower catalog manager: success
I20260927 12:50:44.731413 23289 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 200646d3c99f4626a64f025e81a65073: acquiring CA information for follower catalog manager: success
I20260927 12:50:44.732566 23289 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 200646d3c99f4626a64f025e81a65073: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 12:50:44.748028 23313 catalog_manager.cc:1259] Loaded cluster ID: 64a27e5bb0d2494d88584da4b6aa3fa0
I20260927 12:50:44.748111 23313 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 217eb45bb3e34dd58a4275a4e4bc24d0: loading cluster ID for follower catalog manager: success
I20260927 12:50:44.749739 23313 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 217eb45bb3e34dd58a4275a4e4bc24d0: acquiring CA information for follower catalog manager: success
I20260927 12:50:44.750910 23313 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 217eb45bb3e34dd58a4275a4e4bc24d0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 12:50:44.930878 23390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:44.927259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56138 (local address 127.0.98.61:37335)
0927 12:50:44.927498 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:44.927502 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:44.927578 (+ 76us) server_negotiation.cc:407] Connection header received
0927 12:50:44.927684 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:44.927686 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:44.927832 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:44.927941 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:50:44.928149 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:44.928684 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:44.929521 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:44.929750 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:44.929835 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:44.930100 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:50:44.930526 (+ 426us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:50:44.930611 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:44.930668 (+ 57us) server_negotiation.cc:299] Negotiation successful
0927 12:50:44.930713 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":71,"threads_started":1}
I20260927 12:50:44.937004 23396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:44.933574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33206 (local address 127.0.98.60:41253)
0927 12:50:44.933849 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:44.933853 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:44.933865 (+ 12us) server_negotiation.cc:407] Connection header received
0927 12:50:44.933941 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:44.933944 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:44.934091 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:44.934183 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:50:44.934352 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:44.934873 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:44.935663 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:44.935931 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:44.936010 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:44.936228 (+ 218us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:50:44.936637 (+ 409us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:50:44.936721 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:44.936798 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 12:50:44.936854 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":73,"threads_started":1}
I20260927 12:50:44.943774 23390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:44.939829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56144 (local address 127.0.98.61:37335)
0927 12:50:44.939955 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:44.939959 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:44.940274 (+ 315us) server_negotiation.cc:407] Connection header received
0927 12:50:44.940398 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:44.940401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:44.940449 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:44.940522 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:50:44.940737 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:44.941471 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:44.942624 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:44.942912 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:44.942999 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:44.943270 (+ 271us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:50:44.943352 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:50:44.943416 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:44.943568 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 12:50:44.943624 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 12:50:44.944403 23403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:44.939915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46816 (local address 127.0.98.62:38661)
0927 12:50:44.940153 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:44.940157 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:44.940174 (+ 17us) server_negotiation.cc:407] Connection header received
0927 12:50:44.940266 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:44.940269 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:44.940428 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:44.940535 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:50:44.941187 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:44.941842 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:44.942781 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:44.943144 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:44.943249 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:44.943596 (+ 347us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:50:44.943958 (+ 362us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:50:44.944032 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:44.944180 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 12:50:44.944252 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":69,"threads_started":1}
I20260927 12:50:44.944860 23396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:50:44.940338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33214 (local address 127.0.98.60:41253)
0927 12:50:44.940435 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0927 12:50:44.940438 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:50:44.940816 (+ 378us) server_negotiation.cc:407] Connection header received
0927 12:50:44.940956 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:50:44.940959 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:50:44.941003 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:50:44.941085 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:50:44.941345 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:44.942084 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:44.943288 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:50:44.943583 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:50:44.943666 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:50:44.943982 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:50:44.944053 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:50:44.944095 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0927 12:50:44.944701 (+ 606us) server_negotiation.cc:299] Negotiation successful
0927 12:50:44.944768 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 12:50:44.947489 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 23038
I20260927 12:50:44.953402 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 23113
I20260927 12:50:44.958155 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTC8Et9/build/debug/bin/kudu with pid 23188
2026-09-27T12:50:44Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2977 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6030 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (283789 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST